Note: This is test shard 8 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from TableRangeRebalancingTest
[ RUN ] TableRangeRebalancingTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 02:11:23.311141 23971 test_util.cc:268] Using random seed: 1014892530
I20240503 02:11:23.322154 23971 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:11:23.322225 23971 ts_itest-base.cc:113] --------------
I20240503 02:11:23.322261 23971 ts_itest-base.cc:114] 3 tablet servers
I20240503 02:11:23.322299 23971 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:11:23.322332 23971 ts_itest-base.cc:116] --------------
2024-05-03T02:11:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:11:23Z Disabled control of system clock
I20240503 02:11:23.344740 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.104.254:41945
--webserver_interface=127.23.104.254
--webserver_port=0
--builtin_ntp_servers=127.23.104.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.104.254:41945
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 02:11:23.648773 23983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:23.649048 23983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:23.649135 23983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:23.656216 23983 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:23.656325 23983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:11:23.656378 23983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:23.656433 23983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:11:23.656481 23983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:11:23.665342 23983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.23.104.254:41945
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.104.254:41945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.23.104.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:23.666579 23983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:23.668241 23983 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:23.678689 23989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:23.679420 23988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:23.680037 23991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:23.680759 23983 server_base.cc:1034] running on GCE node
I20240503 02:11:23.681409 23983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:23.682695 23983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:23.683864 23983 hybrid_clock.cc:648] HybridClock initialized: now 1714702283683826 us; error 27 us; skew 500 ppm
I20240503 02:11:23.684190 23983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:23.686262 23983 webserver.cc:460] Webserver started at http://127.23.104.254:46407/ using document root <none> and password file <none>
I20240503 02:11:23.686924 23983 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:23.687038 23983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:23.687331 23983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:23.689774 23983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0fb065e16cb345efa68470d06f599dcb"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-38ht"
I20240503 02:11:23.690479 23983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0fb065e16cb345efa68470d06f599dcb"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-38ht"
I20240503 02:11:23.695067 23983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:11:23.698104 23997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:23.699211 23983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 02:11:23.699366 23983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0fb065e16cb345efa68470d06f599dcb"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-38ht"
I20240503 02:11:23.699564 23983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:23.711545 23983 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:23.712301 23983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:23.712577 23983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:23.733253 23983 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.254:41945
I20240503 02:11:23.733282 24048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.254:41945 every 8 connection(s)
I20240503 02:11:23.734391 23983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:11:23.737545 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 23983
I20240503 02:11:23.737797 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:11:23.737905 24049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:23.747445 24049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb: Bootstrap starting.
I20240503 02:11:23.750027 24049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:23.750910 24049 log.cc:824] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:23.753357 24049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb: No bootstrap required, opened a new log
I20240503 02:11:23.759035 24049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb065e16cb345efa68470d06f599dcb" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 41945 } }
I20240503 02:11:23.759343 24049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:23.759449 24049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fb065e16cb345efa68470d06f599dcb, State: Initialized, Role: FOLLOWER
I20240503 02:11:23.760143 24049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb065e16cb345efa68470d06f599dcb" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 41945 } }
I20240503 02:11:23.760460 24049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:23.760608 24049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:23.760829 24049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:23.762949 24049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb065e16cb345efa68470d06f599dcb" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 41945 } }
I20240503 02:11:23.763487 24049 leader_election.cc:302] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fb065e16cb345efa68470d06f599dcb; no voters:
I20240503 02:11:23.764096 24049 leader_election.cc:288] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:23.764212 24054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:23.764793 24054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [term 1 LEADER]: Becoming Leader. State: Replica: 0fb065e16cb345efa68470d06f599dcb, State: Running, Role: LEADER
I20240503 02:11:23.765308 24054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb065e16cb345efa68470d06f599dcb" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 41945 } }
I20240503 02:11:23.765830 24049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:23.768106 24056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fb065e16cb345efa68470d06f599dcb. Latest consensus state: current_term: 1 leader_uuid: "0fb065e16cb345efa68470d06f599dcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb065e16cb345efa68470d06f599dcb" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 41945 } } }
I20240503 02:11:23.768401 24056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:23.768193 24055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fb065e16cb345efa68470d06f599dcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb065e16cb345efa68470d06f599dcb" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 41945 } } }
I20240503 02:11:23.768563 24055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:23.769866 24059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:23.774720 24059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:23.782982 24059 catalog_manager.cc:1347] Generated new cluster ID: 5548fc0604b449cab2841f9101a98b8a
I20240503 02:11:23.783094 24059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:23.805222 24059 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:23.806923 24059 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:23.822355 24059 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0fb065e16cb345efa68470d06f599dcb: Generated new TSK 0
I20240503 02:11:23.823383 24059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:23.831967 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.193:0
--local_ip_for_outbound_sockets=127.23.104.193
--webserver_interface=127.23.104.193
--webserver_port=0
--tserver_master_addrs=127.23.104.254:41945
--builtin_ntp_servers=127.23.104.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:11:24.138820 24073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:24.139060 24073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:24.139205 24073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:24.146071 24073 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:24.146188 24073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:24.146353 24073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.193
I20240503 02:11:24.154265 24073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.23.104.193
--webserver_port=0
--tserver_master_addrs=127.23.104.254:41945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.193
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:24.155331 24073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:24.157027 24073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:24.168100 24078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.168712 24079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.169284 24081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:24.169924 24073 server_base.cc:1034] running on GCE node
I20240503 02:11:24.170389 24073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:24.171487 24073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:24.172690 24073 hybrid_clock.cc:648] HybridClock initialized: now 1714702284172640 us; error 43 us; skew 500 ppm
I20240503 02:11:24.172959 24073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:24.174840 24073 webserver.cc:460] Webserver started at http://127.23.104.193:34627/ using document root <none> and password file <none>
I20240503 02:11:24.175444 24073 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:24.175544 24073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:24.175892 24073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:24.178289 24073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "bd6be00943a94f71aec0a4dd88593369"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-38ht"
I20240503 02:11:24.179005 24073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "bd6be00943a94f71aec0a4dd88593369"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-38ht"
I20240503 02:11:24.183516 24073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240503 02:11:24.186620 24087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:24.187820 24073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:24.187960 24073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bd6be00943a94f71aec0a4dd88593369"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-38ht"
I20240503 02:11:24.188129 24073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:24.200254 24073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:24.200927 24073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:24.201167 24073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:24.201973 24073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:24.203542 24073 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:24.203645 24073 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:24.203761 24073 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:24.203860 24073 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240503 02:11:24.238376 24073 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.193:38079
I20240503 02:11:24.238420 24199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.193:38079 every 8 connection(s)
I20240503 02:11:24.239681 24073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:11:24.246400 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 24073
I20240503 02:11:24.246593 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:11:24.249840 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.194:0
--local_ip_for_outbound_sockets=127.23.104.194
--webserver_interface=127.23.104.194
--webserver_port=0
--tserver_master_addrs=127.23.104.254:41945
--builtin_ntp_servers=127.23.104.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:24.251467 24200 heartbeater.cc:347] Connected to a master server at 127.23.104.254:41945
I20240503 02:11:24.251721 24200 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:24.252560 24200 heartbeater.cc:511] Master 127.23.104.254:41945 requested a full tablet report, sending...
I20240503 02:11:24.254676 24014 ts_manager.cc:194] Registered new tserver with Master: bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079)
I20240503 02:11:24.256853 24014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.193:54437
W20240503 02:11:24.559593 24203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:24.559893 24203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:24.559974 24203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:24.567481 24203 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:24.567586 24203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:24.567704 24203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.194
I20240503 02:11:24.576267 24203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.23.104.194
--webserver_port=0
--tserver_master_addrs=127.23.104.254:41945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.194
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:24.577415 24203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:24.579128 24203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:24.591122 24208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.591774 24211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.592291 24209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:24.592989 24203 server_base.cc:1034] running on GCE node
I20240503 02:11:24.593518 24203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:24.594359 24203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:24.595558 24203 hybrid_clock.cc:648] HybridClock initialized: now 1714702284595512 us; error 40 us; skew 500 ppm
I20240503 02:11:24.595881 24203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:24.598416 24203 webserver.cc:460] Webserver started at http://127.23.104.194:46189/ using document root <none> and password file <none>
I20240503 02:11:24.599081 24203 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:24.599196 24203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:24.599488 24203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:24.601882 24203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "844d6da45edb49e19a8c621b067cc515"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-38ht"
I20240503 02:11:24.602564 24203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "844d6da45edb49e19a8c621b067cc515"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-38ht"
I20240503 02:11:24.607214 24203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20240503 02:11:24.610378 24217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:24.611519 24203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:24.611675 24203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "844d6da45edb49e19a8c621b067cc515"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-38ht"
I20240503 02:11:24.611882 24203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:24.638310 24203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:24.639058 24203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:24.639312 24203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:24.640233 24203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:24.641516 24203 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:24.641616 24203 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:24.641729 24203 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:24.641799 24203 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:24.673940 24203 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.194:46841
I20240503 02:11:24.673995 24329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.194:46841 every 8 connection(s)
I20240503 02:11:24.675187 24203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:11:24.676494 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 24203
I20240503 02:11:24.676666 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:11:24.680289 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.195:0
--local_ip_for_outbound_sockets=127.23.104.195
--webserver_interface=127.23.104.195
--webserver_port=0
--tserver_master_addrs=127.23.104.254:41945
--builtin_ntp_servers=127.23.104.212:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:24.688475 24330 heartbeater.cc:347] Connected to a master server at 127.23.104.254:41945
I20240503 02:11:24.688781 24330 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:24.689765 24330 heartbeater.cc:511] Master 127.23.104.254:41945 requested a full tablet report, sending...
I20240503 02:11:24.691303 24014 ts_manager.cc:194] Registered new tserver with Master: 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841)
I20240503 02:11:24.692078 24014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.194:42951
W20240503 02:11:25.001716 24333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:25.001983 24333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:25.002113 24333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:25.010516 24333 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:25.010636 24333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:25.010757 24333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.195
I20240503 02:11:25.020629 24333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.23.104.195
--webserver_port=0
--tserver_master_addrs=127.23.104.254:41945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.195
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:25.021922 24333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:25.023583 24333 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:25.035771 24338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:25.036314 24339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:25.036935 24341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:25.037590 24333 server_base.cc:1034] running on GCE node
I20240503 02:11:25.038123 24333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:25.039008 24333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:25.040268 24333 hybrid_clock.cc:648] HybridClock initialized: now 1714702285040182 us; error 72 us; skew 500 ppm
I20240503 02:11:25.040596 24333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:25.042610 24333 webserver.cc:460] Webserver started at http://127.23.104.195:41521/ using document root <none> and password file <none>
I20240503 02:11:25.043280 24333 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:25.043397 24333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:25.043694 24333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:25.049878 24333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7b26dff28ae1498e85be69e29aa7a827"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-38ht"
I20240503 02:11:25.050534 24333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7b26dff28ae1498e85be69e29aa7a827"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-38ht"
I20240503 02:11:25.055141 24333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:11:25.058213 24347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.059391 24333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:25.059538 24333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7b26dff28ae1498e85be69e29aa7a827"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-38ht"
I20240503 02:11:25.059741 24333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:25.097572 24333 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:25.098376 24333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:25.098670 24333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:25.099546 24333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:25.100917 24333 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:25.101011 24333 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.101126 24333 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:25.101195 24333 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.134588 24333 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.195:33273
I20240503 02:11:25.134644 24459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.195:33273 every 8 connection(s)
I20240503 02:11:25.135890 24333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:11:25.139266 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 24333
I20240503 02:11:25.139452 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:11:25.148175 24460 heartbeater.cc:347] Connected to a master server at 127.23.104.254:41945
I20240503 02:11:25.148430 24460 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:25.149168 24460 heartbeater.cc:511] Master 127.23.104.254:41945 requested a full tablet report, sending...
I20240503 02:11:25.150290 24014 ts_manager.cc:194] Registered new tserver with Master: 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:25.150993 24014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.195:44379
I20240503 02:11:25.155061 23971 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 02:11:25.173156 23971 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 02:11:25.259503 24200 heartbeater.cc:503] Master 127.23.104.254:41945 was elected leader, sending a full tablet report...
I20240503 02:11:25.694811 24330 heartbeater.cc:503] Master 127.23.104.254:41945 was elected leader, sending a full tablet report...
I20240503 02:11:26.153050 24460 heartbeater.cc:503] Master 127.23.104.254:41945 was elected leader, sending a full tablet report...
I20240503 02:11:28.927114 24014 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34304:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\200\007\001\000\000\000\000\000\006\001\000\000\000\000\000\007\001\000\377\377\377\177"
indirect_data: "<redacted>"""
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 8
seed: 0
}
}
W20240503 02:11:28.933589 24014 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:11:28.979856 24392 tablet_service.cc:1467] Processing CreateTablet for tablet 7317ad2eec8d42f4bd620fc62d904325 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.981582 24381 tablet_service.cc:1467] Processing CreateTablet for tablet b1fa9034b37c4374b469e3f1e5b9cda1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:28.981055 24380 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5e5a2517bd48b7a58cee8cb27facd2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.982456 24382 tablet_service.cc:1467] Processing CreateTablet for tablet 2e94ed8ccadf4b22bbe16211a7dfab4c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.982775 24380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5e5a2517bd48b7a58cee8cb27facd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.983282 24391 tablet_service.cc:1467] Processing CreateTablet for tablet 0f9b3e7e2bc447e69c138cd3a8c78f29 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:28.984202 24390 tablet_service.cc:1467] Processing CreateTablet for tablet f16b4870e2d34ba89ca7469266f7857b (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.984982 24389 tablet_service.cc:1467] Processing CreateTablet for tablet c991b40002b14c6ba8aaf6ab920f26d4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:28.985699 24388 tablet_service.cc:1467] Processing CreateTablet for tablet dbb09cbbb3604a698492e47e3e61cfcb (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.986438 24386 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc668713bbe4907a612df4c29acbe7d (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.986450 24265 tablet_service.cc:1467] Processing CreateTablet for tablet 42b7394682e342fd93c46e68e8b549bd (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:28.988077 24265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42b7394682e342fd93c46e68e8b549bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.988219 24384 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd69f29c6854595845edfbc98867aa5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.988824 24391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9b3e7e2bc447e69c138cd3a8c78f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.989092 24392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7317ad2eec8d42f4bd620fc62d904325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.989899 24128 tablet_service.cc:1467] Processing CreateTablet for tablet dbb09cbbb3604a698492e47e3e61cfcb (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.991113 24263 tablet_service.cc:1467] Processing CreateTablet for tablet 3ed83724e7f241a3ae68193b1e168b4c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:28.991509 24128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb09cbbb3604a698492e47e3e61cfcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.991621 24263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ed83724e7f241a3ae68193b1e168b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.991927 24388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb09cbbb3604a698492e47e3e61cfcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.992355 24382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e94ed8ccadf4b22bbe16211a7dfab4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.992604 24261 tablet_service.cc:1467] Processing CreateTablet for tablet 0f9b3e7e2bc447e69c138cd3a8c78f29 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:28.992856 24254 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd69f29c6854595845edfbc98867aa5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.993228 24261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9b3e7e2bc447e69c138cd3a8c78f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.993520 24250 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5e5a2517bd48b7a58cee8cb27facd2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.993772 24264 tablet_service.cc:1467] Processing CreateTablet for tablet 3173cd7821d6431388828d6e40627791 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.994354 24254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd69f29c6854595845edfbc98867aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.994900 24389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c991b40002b14c6ba8aaf6ab920f26d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.995177 24250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5e5a2517bd48b7a58cee8cb27facd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.995208 24394 tablet_service.cc:1467] Processing CreateTablet for tablet 3173cd7821d6431388828d6e40627791 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.994688 24393 tablet_service.cc:1467] Processing CreateTablet for tablet 3ed83724e7f241a3ae68193b1e168b4c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:28.996872 24260 tablet_service.cc:1467] Processing CreateTablet for tablet f16b4870e2d34ba89ca7469266f7857b (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.996932 24253 tablet_service.cc:1467] Processing CreateTablet for tablet 3483d66d5e0b40928016b55c481fdfbc (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:28.997519 24262 tablet_service.cc:1467] Processing CreateTablet for tablet 7317ad2eec8d42f4bd620fc62d904325 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.997725 24251 tablet_service.cc:1467] Processing CreateTablet for tablet b1fa9034b37c4374b469e3f1e5b9cda1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:28.997797 24252 tablet_service.cc:1467] Processing CreateTablet for tablet 2e94ed8ccadf4b22bbe16211a7dfab4c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.997915 24255 tablet_service.cc:1467] Processing CreateTablet for tablet b16a60476daa4740b32781ecf3b1554d (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:28.994436 24259 tablet_service.cc:1467] Processing CreateTablet for tablet c991b40002b14c6ba8aaf6ab920f26d4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:28.996630 24126 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc668713bbe4907a612df4c29acbe7d (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.998808 24257 tablet_service.cc:1467] Processing CreateTablet for tablet 3c9f1118b6694b88a82dcbbf34c5d8a5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:28.998847 24120 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5e5a2517bd48b7a58cee8cb27facd2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:28.999392 24126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc668713bbe4907a612df4c29acbe7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.999454 24260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16b4870e2d34ba89ca7469266f7857b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.000260 24120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5e5a2517bd48b7a58cee8cb27facd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.000320 24127 tablet_service.cc:1467] Processing CreateTablet for tablet 3c9f1118b6694b88a82dcbbf34c5d8a5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:29.000376 24124 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd69f29c6854595845edfbc98867aa5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:29.000866 24127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9f1118b6694b88a82dcbbf34c5d8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.001351 24383 tablet_service.cc:1467] Processing CreateTablet for tablet 3483d66d5e0b40928016b55c481fdfbc (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:29.001392 24395 tablet_service.cc:1467] Processing CreateTablet for tablet 42b7394682e342fd93c46e68e8b549bd (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:29.001277 24129 tablet_service.cc:1467] Processing CreateTablet for tablet c991b40002b14c6ba8aaf6ab920f26d4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:29.000257 24131 tablet_service.cc:1467] Processing CreateTablet for tablet 0f9b3e7e2bc447e69c138cd3a8c78f29 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:29.002553 24124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd69f29c6854595845edfbc98867aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.003078 24257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9f1118b6694b88a82dcbbf34c5d8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.003466 24264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3173cd7821d6431388828d6e40627791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.002915 24256 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc668713bbe4907a612df4c29acbe7d (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:29.001477 24133 tablet_service.cc:1467] Processing CreateTablet for tablet 3ed83724e7f241a3ae68193b1e168b4c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:29.003755 24262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7317ad2eec8d42f4bd620fc62d904325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.004040 24129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c991b40002b14c6ba8aaf6ab920f26d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.003820 24258 tablet_service.cc:1467] Processing CreateTablet for tablet dbb09cbbb3604a698492e47e3e61cfcb (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:29.004149 24130 tablet_service.cc:1467] Processing CreateTablet for tablet f16b4870e2d34ba89ca7469266f7857b (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:29.005060 24252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e94ed8ccadf4b22bbe16211a7dfab4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.005291 24253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3483d66d5e0b40928016b55c481fdfbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.005509 24255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16a60476daa4740b32781ecf3b1554d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.005738 24258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb09cbbb3604a698492e47e3e61cfcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.005623 24387 tablet_service.cc:1467] Processing CreateTablet for tablet 3c9f1118b6694b88a82dcbbf34c5d8a5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:29.005959 24251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fa9034b37c4374b469e3f1e5b9cda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.004957 24122 tablet_service.cc:1467] Processing CreateTablet for tablet 2e94ed8ccadf4b22bbe16211a7dfab4c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:29.004981 24121 tablet_service.cc:1467] Processing CreateTablet for tablet b1fa9034b37c4374b469e3f1e5b9cda1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:29.006537 24259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c991b40002b14c6ba8aaf6ab920f26d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.007064 24390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16b4870e2d34ba89ca7469266f7857b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.007673 24256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc668713bbe4907a612df4c29acbe7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.007690 24381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fa9034b37c4374b469e3f1e5b9cda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.007427 24385 tablet_service.cc:1467] Processing CreateTablet for tablet b16a60476daa4740b32781ecf3b1554d (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:29.007915 24393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ed83724e7f241a3ae68193b1e168b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.008446 24387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9f1118b6694b88a82dcbbf34c5d8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.008662 24385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16a60476daa4740b32781ecf3b1554d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.009001 24394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3173cd7821d6431388828d6e40627791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.009307 24386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc668713bbe4907a612df4c29acbe7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.009949 24384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd69f29c6854595845edfbc98867aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.010437 24383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3483d66d5e0b40928016b55c481fdfbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.010880 24395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42b7394682e342fd93c46e68e8b549bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.011622 24131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9b3e7e2bc447e69c138cd3a8c78f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.011868 24133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ed83724e7f241a3ae68193b1e168b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.011586 24125 tablet_service.cc:1467] Processing CreateTablet for tablet b16a60476daa4740b32781ecf3b1554d (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:29.011981 24123 tablet_service.cc:1467] Processing CreateTablet for tablet 3483d66d5e0b40928016b55c481fdfbc (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:29.012058 24135 tablet_service.cc:1467] Processing CreateTablet for tablet 42b7394682e342fd93c46e68e8b549bd (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 02:11:29.013428 24132 tablet_service.cc:1467] Processing CreateTablet for tablet 7317ad2eec8d42f4bd620fc62d904325 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:29.014104 24130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16b4870e2d34ba89ca7469266f7857b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.014319 24135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42b7394682e342fd93c46e68e8b549bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.015058 24125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16a60476daa4740b32781ecf3b1554d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.015471 24134 tablet_service.cc:1467] Processing CreateTablet for tablet 3173cd7821d6431388828d6e40627791 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c07420528364484b8b01ae9729b2b036]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 02:11:29.016157 24123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3483d66d5e0b40928016b55c481fdfbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.016530 24132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7317ad2eec8d42f4bd620fc62d904325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.017329 24121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fa9034b37c4374b469e3f1e5b9cda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.017951 24122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e94ed8ccadf4b22bbe16211a7dfab4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.019994 24134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3173cd7821d6431388828d6e40627791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.023568 24478 tablet_bootstrap.cc:492] T 4bc668713bbe4907a612df4c29acbe7d P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.027164 24478 tablet_bootstrap.cc:654] T 4bc668713bbe4907a612df4c29acbe7d P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.028556 24478 log.cc:824] T 4bc668713bbe4907a612df4c29acbe7d P 7b26dff28ae1498e85be69e29aa7a827: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:29.084307 24480 tablet_bootstrap.cc:492] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.098220 24480 tablet_bootstrap.cc:654] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.099112 24478 tablet_bootstrap.cc:492] T 4bc668713bbe4907a612df4c29acbe7d P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.099447 24478 ts_tablet_manager.cc:1367] T 4bc668713bbe4907a612df4c29acbe7d P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.076s user 0.024s sys 0.037s
I20240503 02:11:29.101435 24480 log.cc:824] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:29.108902 24478 raft_consensus.cc:348] T 4bc668713bbe4907a612df4c29acbe7d P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.109517 24478 raft_consensus.cc:374] T 4bc668713bbe4907a612df4c29acbe7d P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.109689 24478 raft_consensus.cc:729] T 4bc668713bbe4907a612df4c29acbe7d P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.110577 24478 consensus_queue.cc:260] T 4bc668713bbe4907a612df4c29acbe7d P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.114715 24480 tablet_bootstrap.cc:492] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.115106 24480 ts_tablet_manager.cc:1367] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.000s
I20240503 02:11:29.118683 24478 ts_tablet_manager.cc:1398] T 4bc668713bbe4907a612df4c29acbe7d P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.019s user 0.011s sys 0.007s
I20240503 02:11:29.119168 24478 tablet_bootstrap.cc:492] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.120885 24478 tablet_bootstrap.cc:654] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.126634 24480 raft_consensus.cc:348] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.127300 24480 raft_consensus.cc:374] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.127467 24480 raft_consensus.cc:729] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.128381 24480 consensus_queue.cc:260] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.132812 24478 tablet_bootstrap.cc:492] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.133004 24478 ts_tablet_manager.cc:1367] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240503 02:11:29.133950 24478 raft_consensus.cc:348] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.134224 24478 raft_consensus.cc:374] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.134346 24478 raft_consensus.cc:729] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.134981 24478 consensus_queue.cc:260] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.135675 24478 ts_tablet_manager.cc:1398] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:11:29.135984 24478 tablet_bootstrap.cc:492] T f16b4870e2d34ba89ca7469266f7857b P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.137674 24478 tablet_bootstrap.cc:654] T f16b4870e2d34ba89ca7469266f7857b P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
W20240503 02:11:29.140015 24461 tablet.cc:2369] T 4bc668713bbe4907a612df4c29acbe7d P 7b26dff28ae1498e85be69e29aa7a827: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:29.181108 24482 tablet_bootstrap.cc:492] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.185972 24482 tablet_bootstrap.cc:654] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.187367 24482 log.cc:824] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:29.187844 24480 ts_tablet_manager.cc:1398] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.073s user 0.044s sys 0.027s
I20240503 02:11:29.188395 24480 tablet_bootstrap.cc:492] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.188622 24478 tablet_bootstrap.cc:492] T f16b4870e2d34ba89ca7469266f7857b P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.188825 24478 ts_tablet_manager.cc:1367] T f16b4870e2d34ba89ca7469266f7857b P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.053s user 0.019s sys 0.032s
I20240503 02:11:29.189632 24478 raft_consensus.cc:348] T f16b4870e2d34ba89ca7469266f7857b P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.189908 24478 raft_consensus.cc:374] T f16b4870e2d34ba89ca7469266f7857b P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.190017 24478 raft_consensus.cc:729] T f16b4870e2d34ba89ca7469266f7857b P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.190436 24480 tablet_bootstrap.cc:654] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.190315 24478 consensus_queue.cc:260] T f16b4870e2d34ba89ca7469266f7857b P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.190904 24478 ts_tablet_manager.cc:1398] T f16b4870e2d34ba89ca7469266f7857b P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:11:29.191247 24478 tablet_bootstrap.cc:492] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.193357 24478 tablet_bootstrap.cc:654] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.197237 24478 tablet_bootstrap.cc:492] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.197369 24478 ts_tablet_manager.cc:1367] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:11:29.198020 24478 raft_consensus.cc:348] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.198284 24478 raft_consensus.cc:374] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.198381 24478 raft_consensus.cc:729] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.198644 24478 consensus_queue.cc:260] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.198846 24480 tablet_bootstrap.cc:492] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.199046 24480 ts_tablet_manager.cc:1367] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240503 02:11:29.199268 24478 ts_tablet_manager.cc:1398] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:11:29.200083 24480 raft_consensus.cc:348] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } }
I20240503 02:11:29.200305 24480 raft_consensus.cc:374] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.199585 24478 tablet_bootstrap.cc:492] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.200470 24480 raft_consensus.cc:729] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.200667 24482 tablet_bootstrap.cc:492] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.200793 24480 consensus_queue.cc:260] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } }
I20240503 02:11:29.200991 24482 ts_tablet_manager.cc:1367] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.000s
I20240503 02:11:29.201471 24480 ts_tablet_manager.cc:1398] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:29.201723 24480 tablet_bootstrap.cc:492] T c991b40002b14c6ba8aaf6ab920f26d4 P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.203264 24480 tablet_bootstrap.cc:654] T c991b40002b14c6ba8aaf6ab920f26d4 P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.204027 24478 tablet_bootstrap.cc:654] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.212797 24482 raft_consensus.cc:348] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.213281 24482 raft_consensus.cc:374] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.213405 24482 raft_consensus.cc:729] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.213945 24478 tablet_bootstrap.cc:492] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.213963 24485 raft_consensus.cc:482] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.214010 24480 tablet_bootstrap.cc:492] T c991b40002b14c6ba8aaf6ab920f26d4 P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.214129 24478 ts_tablet_manager.cc:1367] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.001s
I20240503 02:11:29.214252 24480 ts_tablet_manager.cc:1367] T c991b40002b14c6ba8aaf6ab920f26d4 P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20240503 02:11:29.214144 24482 consensus_queue.cc:260] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.214226 24485 raft_consensus.cc:504] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.214903 24478 raft_consensus.cc:348] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.215096 24478 raft_consensus.cc:374] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.215162 24478 raft_consensus.cc:729] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.215193 24480 raft_consensus.cc:348] T c991b40002b14c6ba8aaf6ab920f26d4 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.215418 24478 consensus_queue.cc:260] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.215586 24480 raft_consensus.cc:374] T c991b40002b14c6ba8aaf6ab920f26d4 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.215679 24480 raft_consensus.cc:729] T c991b40002b14c6ba8aaf6ab920f26d4 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.216053 24478 ts_tablet_manager.cc:1398] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:29.216059 24480 consensus_queue.cc:260] T c991b40002b14c6ba8aaf6ab920f26d4 P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.216374 24485 leader_election.cc:288] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.220211 24478 tablet_bootstrap.cc:492] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.222153 24478 tablet_bootstrap.cc:654] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.223496 24480 ts_tablet_manager.cc:1398] T c991b40002b14c6ba8aaf6ab920f26d4 P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240503 02:11:29.223909 24480 tablet_bootstrap.cc:492] T 3483d66d5e0b40928016b55c481fdfbc P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.225739 24480 tablet_bootstrap.cc:654] T 3483d66d5e0b40928016b55c481fdfbc P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.238879 24482 ts_tablet_manager.cc:1398] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.038s user 0.015s sys 0.004s
I20240503 02:11:29.239348 24482 tablet_bootstrap.cc:492] T 6cd69f29c6854595845edfbc98867aa5 P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.240943 24482 tablet_bootstrap.cc:654] T 6cd69f29c6854595845edfbc98867aa5 P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
W20240503 02:11:29.244320 24201 tablet.cc:2369] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:29.245543 24285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc668713bbe4907a612df4c29acbe7d" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
W20240503 02:11:29.246953 24090 leader_election.cc:341] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.249466 24478 tablet_bootstrap.cc:492] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.249361 24415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc668713bbe4907a612df4c29acbe7d" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827" is_pre_election: true
I20240503 02:11:29.249635 24478 ts_tablet_manager.cc:1367] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.009s
I20240503 02:11:29.249930 24415 raft_consensus.cc:2454] T 4bc668713bbe4907a612df4c29acbe7d P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd6be00943a94f71aec0a4dd88593369 in term 0.
I20240503 02:11:29.250422 24478 raft_consensus.cc:348] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.250589 24478 raft_consensus.cc:374] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.250665 24478 raft_consensus.cc:729] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.250877 24478 consensus_queue.cc:260] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.251308 24478 ts_tablet_manager.cc:1398] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:11:29.251267 24089 leader_election.cc:302] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters: 844d6da45edb49e19a8c621b067cc515
I20240503 02:11:29.251541 24478 tablet_bootstrap.cc:492] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.251825 24493 raft_consensus.cc:2781] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.251955 24493 raft_consensus.cc:482] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.252095 24493 raft_consensus.cc:3037] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.253217 24478 tablet_bootstrap.cc:654] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.255422 24480 tablet_bootstrap.cc:492] T 3483d66d5e0b40928016b55c481fdfbc P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.255563 24480 ts_tablet_manager.cc:1367] T 3483d66d5e0b40928016b55c481fdfbc P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.012s
I20240503 02:11:29.255664 24493 raft_consensus.cc:504] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.256435 24482 tablet_bootstrap.cc:492] T 6cd69f29c6854595845edfbc98867aa5 P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.256337 24480 raft_consensus.cc:348] T 3483d66d5e0b40928016b55c481fdfbc P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.256531 24480 raft_consensus.cc:374] T 3483d66d5e0b40928016b55c481fdfbc P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.256579 24482 ts_tablet_manager.cc:1367] T 6cd69f29c6854595845edfbc98867aa5 P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240503 02:11:29.256668 24480 raft_consensus.cc:729] T 3483d66d5e0b40928016b55c481fdfbc P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.256930 24480 consensus_queue.cc:260] T 3483d66d5e0b40928016b55c481fdfbc P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.257483 24480 ts_tablet_manager.cc:1398] T 3483d66d5e0b40928016b55c481fdfbc P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:29.257457 24415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc668713bbe4907a612df4c29acbe7d" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827"
I20240503 02:11:29.257769 24480 tablet_bootstrap.cc:492] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.257697 24415 raft_consensus.cc:3037] T 4bc668713bbe4907a612df4c29acbe7d P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.259407 24480 tablet_bootstrap.cc:654] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.257156 24285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc668713bbe4907a612df4c29acbe7d" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515"
W20240503 02:11:29.261842 24090 leader_election.cc:341] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.262782 24415 raft_consensus.cc:2454] T 4bc668713bbe4907a612df4c29acbe7d P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd6be00943a94f71aec0a4dd88593369 in term 1.
I20240503 02:11:29.263031 24493 leader_election.cc:288] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Requested vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.263217 24482 raft_consensus.cc:348] T 6cd69f29c6854595845edfbc98867aa5 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.263471 24482 raft_consensus.cc:374] T 6cd69f29c6854595845edfbc98867aa5 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.263567 24482 raft_consensus.cc:729] T 6cd69f29c6854595845edfbc98867aa5 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.263724 24089 leader_election.cc:302] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters: 844d6da45edb49e19a8c621b067cc515
I20240503 02:11:29.263945 24482 consensus_queue.cc:260] T 6cd69f29c6854595845edfbc98867aa5 P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.264417 24493 raft_consensus.cc:2781] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.264863 24493 raft_consensus.cc:686] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [term 1 LEADER]: Becoming Leader. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Running, Role: LEADER
I20240503 02:11:29.265689 24493 consensus_queue.cc:237] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.267684 24482 ts_tablet_manager.cc:1398] T 6cd69f29c6854595845edfbc98867aa5 P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:11:29.270480 24482 tablet_bootstrap.cc:492] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.272675 24482 tablet_bootstrap.cc:654] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.274964 24482 tablet_bootstrap.cc:492] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.275157 24482 ts_tablet_manager.cc:1367] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:11:29.276008 24482 raft_consensus.cc:348] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.276293 24482 raft_consensus.cc:374] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.276386 24482 raft_consensus.cc:729] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.276908 24482 consensus_queue.cc:260] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.277580 24478 tablet_bootstrap.cc:492] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.277817 24478 ts_tablet_manager.cc:1367] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.011s
I20240503 02:11:29.278576 24478 raft_consensus.cc:348] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.278841 24478 raft_consensus.cc:374] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.278968 24478 raft_consensus.cc:729] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.279320 24478 consensus_queue.cc:260] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.279922 24478 ts_tablet_manager.cc:1398] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:11:29.280936 24478 tablet_bootstrap.cc:492] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.282593 24478 tablet_bootstrap.cc:654] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.280874 24482 ts_tablet_manager.cc:1398] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:11:29.288463 24482 tablet_bootstrap.cc:492] T dbb09cbbb3604a698492e47e3e61cfcb P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.291684 24478 tablet_bootstrap.cc:492] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.292074 24478 ts_tablet_manager.cc:1367] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20240503 02:11:29.292860 24478 raft_consensus.cc:348] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.293143 24478 raft_consensus.cc:374] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.293273 24478 raft_consensus.cc:729] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.292886 24482 tablet_bootstrap.cc:654] T dbb09cbbb3604a698492e47e3e61cfcb P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.293582 24478 consensus_queue.cc:260] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.294252 24478 ts_tablet_manager.cc:1398] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:11:29.294715 24478 tablet_bootstrap.cc:492] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.294587 24012 catalog_manager.cc:5484] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd6be00943a94f71aec0a4dd88593369 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "bd6be00943a94f71aec0a4dd88593369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:29.297544 24478 tablet_bootstrap.cc:654] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.298588 24480 tablet_bootstrap.cc:492] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.298805 24480 ts_tablet_manager.cc:1367] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.041s user 0.017s sys 0.020s
I20240503 02:11:29.299530 24480 raft_consensus.cc:348] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.299769 24480 raft_consensus.cc:374] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.299865 24480 raft_consensus.cc:729] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.300148 24482 tablet_bootstrap.cc:492] T dbb09cbbb3604a698492e47e3e61cfcb P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.300155 24478 tablet_bootstrap.cc:492] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.300107 24480 consensus_queue.cc:260] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.300284 24482 ts_tablet_manager.cc:1367] T dbb09cbbb3604a698492e47e3e61cfcb P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20240503 02:11:29.300288 24478 ts_tablet_manager.cc:1367] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:11:29.300565 24480 ts_tablet_manager.cc:1398] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:11:29.300935 24480 tablet_bootstrap.cc:492] T b16a60476daa4740b32781ecf3b1554d P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.300895 24478 raft_consensus.cc:348] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.301100 24478 raft_consensus.cc:374] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.301173 24478 raft_consensus.cc:729] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.301412 24478 consensus_queue.cc:260] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.301990 24478 ts_tablet_manager.cc:1398] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:29.302341 24478 tablet_bootstrap.cc:492] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.304339 24478 tablet_bootstrap.cc:654] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.301625 24482 raft_consensus.cc:348] T dbb09cbbb3604a698492e47e3e61cfcb P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.305706 24482 raft_consensus.cc:374] T dbb09cbbb3604a698492e47e3e61cfcb P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.305802 24482 raft_consensus.cc:729] T dbb09cbbb3604a698492e47e3e61cfcb P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.305938 24478 tablet_bootstrap.cc:492] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.306058 24478 ts_tablet_manager.cc:1367] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:11:29.306006 24482 consensus_queue.cc:260] T dbb09cbbb3604a698492e47e3e61cfcb P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.306617 24478 raft_consensus.cc:348] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.306792 24478 raft_consensus.cc:374] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.306907 24478 raft_consensus.cc:729] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.306917 24482 ts_tablet_manager.cc:1398] T dbb09cbbb3604a698492e47e3e61cfcb P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:11:29.307160 24482 tablet_bootstrap.cc:492] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.307169 24478 consensus_queue.cc:260] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.307818 24478 ts_tablet_manager.cc:1398] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:11:29.302489 24480 tablet_bootstrap.cc:654] T b16a60476daa4740b32781ecf3b1554d P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.308107 24478 tablet_bootstrap.cc:492] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.309347 24478 tablet_bootstrap.cc:654] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.310333 24482 tablet_bootstrap.cc:654] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.312247 24480 tablet_bootstrap.cc:492] T b16a60476daa4740b32781ecf3b1554d P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.312351 24480 ts_tablet_manager.cc:1367] T b16a60476daa4740b32781ecf3b1554d P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:11:29.312415 24478 tablet_bootstrap.cc:492] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.312551 24478 ts_tablet_manager.cc:1367] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240503 02:11:29.312876 24480 raft_consensus.cc:348] T b16a60476daa4740b32781ecf3b1554d P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.313005 24480 raft_consensus.cc:374] T b16a60476daa4740b32781ecf3b1554d P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.313069 24480 raft_consensus.cc:729] T b16a60476daa4740b32781ecf3b1554d P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.313064 24478 raft_consensus.cc:348] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.313282 24478 raft_consensus.cc:374] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.313285 24480 consensus_queue.cc:260] T b16a60476daa4740b32781ecf3b1554d P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.313349 24478 raft_consensus.cc:729] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.313560 24478 consensus_queue.cc:260] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.313818 24480 ts_tablet_manager.cc:1398] T b16a60476daa4740b32781ecf3b1554d P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:11:29.313915 24478 ts_tablet_manager.cc:1398] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 02:11:29.314085 24480 tablet_bootstrap.cc:492] T 3ed83724e7f241a3ae68193b1e168b4c P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.314181 24478 tablet_bootstrap.cc:492] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.315359 24478 tablet_bootstrap.cc:654] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.315557 24480 tablet_bootstrap.cc:654] T 3ed83724e7f241a3ae68193b1e168b4c P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.318303 24482 tablet_bootstrap.cc:492] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.318657 24482 ts_tablet_manager.cc:1367] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 02:11:29.320266 24483 raft_consensus.cc:482] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.320387 24482 raft_consensus.cc:348] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.320560 24482 raft_consensus.cc:374] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.320624 24482 raft_consensus.cc:729] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.320822 24482 consensus_queue.cc:260] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.320453 24483 raft_consensus.cc:504] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.321313 24482 ts_tablet_manager.cc:1398] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:11:29.321555 24482 tablet_bootstrap.cc:492] T 4bc668713bbe4907a612df4c29acbe7d P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.322930 24483 leader_election.cc:288] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079)
I20240503 02:11:29.324302 24480 tablet_bootstrap.cc:492] T 3ed83724e7f241a3ae68193b1e168b4c P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.324299 24478 tablet_bootstrap.cc:492] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.324431 24478 ts_tablet_manager.cc:1367] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20240503 02:11:29.324908 24480 ts_tablet_manager.cc:1367] T 3ed83724e7f241a3ae68193b1e168b4c P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240503 02:11:29.325172 24478 raft_consensus.cc:348] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.325371 24478 raft_consensus.cc:374] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.325474 24478 raft_consensus.cc:729] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.325606 24480 raft_consensus.cc:348] T 3ed83724e7f241a3ae68193b1e168b4c P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.325851 24480 raft_consensus.cc:374] T 3ed83724e7f241a3ae68193b1e168b4c P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.325971 24480 raft_consensus.cc:729] T 3ed83724e7f241a3ae68193b1e168b4c P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.326229 24482 tablet_bootstrap.cc:654] T 4bc668713bbe4907a612df4c29acbe7d P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.325909 24478 consensus_queue.cc:260] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.326287 24480 consensus_queue.cc:260] T 3ed83724e7f241a3ae68193b1e168b4c P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.326814 24478 ts_tablet_manager.cc:1398] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:29.327172 24478 tablet_bootstrap.cc:492] T 42b7394682e342fd93c46e68e8b549bd P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.327913 24480 ts_tablet_manager.cc:1398] T 3ed83724e7f241a3ae68193b1e168b4c P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:11:29.328349 24480 tablet_bootstrap.cc:492] T b1fa9034b37c4374b469e3f1e5b9cda1 P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.329023 24478 tablet_bootstrap.cc:654] T 42b7394682e342fd93c46e68e8b549bd P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.330646 24480 tablet_bootstrap.cc:654] T b1fa9034b37c4374b469e3f1e5b9cda1 P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.334432 24480 tablet_bootstrap.cc:492] T b1fa9034b37c4374b469e3f1e5b9cda1 P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.334647 24480 ts_tablet_manager.cc:1367] T b1fa9034b37c4374b469e3f1e5b9cda1 P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240503 02:11:29.336107 24480 raft_consensus.cc:348] T b1fa9034b37c4374b469e3f1e5b9cda1 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.336408 24480 raft_consensus.cc:374] T b1fa9034b37c4374b469e3f1e5b9cda1 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.336548 24480 raft_consensus.cc:729] T b1fa9034b37c4374b469e3f1e5b9cda1 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.336855 24480 consensus_queue.cc:260] T b1fa9034b37c4374b469e3f1e5b9cda1 P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.337730 24480 ts_tablet_manager.cc:1398] T b1fa9034b37c4374b469e3f1e5b9cda1 P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:29.338109 24480 tablet_bootstrap.cc:492] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.340659 24480 tablet_bootstrap.cc:654] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.343294 24478 tablet_bootstrap.cc:492] T 42b7394682e342fd93c46e68e8b549bd P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.343477 24478 ts_tablet_manager.cc:1367] T 42b7394682e342fd93c46e68e8b549bd P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.014s
I20240503 02:11:29.344233 24478 raft_consensus.cc:348] T 42b7394682e342fd93c46e68e8b549bd P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } }
I20240503 02:11:29.344514 24478 raft_consensus.cc:374] T 42b7394682e342fd93c46e68e8b549bd P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.344631 24478 raft_consensus.cc:729] T 42b7394682e342fd93c46e68e8b549bd P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.344924 24478 consensus_queue.cc:260] T 42b7394682e342fd93c46e68e8b549bd P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } }
I20240503 02:11:29.345333 24482 tablet_bootstrap.cc:492] T 4bc668713bbe4907a612df4c29acbe7d P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.345573 24478 ts_tablet_manager.cc:1398] T 42b7394682e342fd93c46e68e8b549bd P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:11:29.345914 24478 tablet_bootstrap.cc:492] T 7317ad2eec8d42f4bd620fc62d904325 P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.347524 24478 tablet_bootstrap.cc:654] T 7317ad2eec8d42f4bd620fc62d904325 P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.347774 24482 ts_tablet_manager.cc:1367] T 4bc668713bbe4907a612df4c29acbe7d P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.013s
I20240503 02:11:29.349399 24482 raft_consensus.cc:348] T 4bc668713bbe4907a612df4c29acbe7d P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.351953 24482 raft_consensus.cc:374] T 4bc668713bbe4907a612df4c29acbe7d P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.352385 24482 raft_consensus.cc:729] T 4bc668713bbe4907a612df4c29acbe7d P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.352331 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed83724e7f241a3ae68193b1e168b4c" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.352845 24155 raft_consensus.cc:2454] T 3ed83724e7f241a3ae68193b1e168b4c P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 0.
I20240503 02:11:29.353331 24483 raft_consensus.cc:482] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.353508 24483 raft_consensus.cc:504] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.353878 24285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed83724e7f241a3ae68193b1e168b4c" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
I20240503 02:11:29.354259 24483 leader_election.cc:288] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841)
I20240503 02:11:29.355129 24351 leader_election.cc:302] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.355415 24285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3483d66d5e0b40928016b55c481fdfbc" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
W20240503 02:11:29.355747 24350 leader_election.cc:341] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.356104 24482 consensus_queue.cc:260] T 4bc668713bbe4907a612df4c29acbe7d P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
W20240503 02:11:29.356487 24350 leader_election.cc:341] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.356786 24483 raft_consensus.cc:2781] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.356889 24483 raft_consensus.cc:482] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.357000 24483 raft_consensus.cc:3037] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.357053 24482 ts_tablet_manager.cc:1398] T 4bc668713bbe4907a612df4c29acbe7d P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:11:29.359058 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3483d66d5e0b40928016b55c481fdfbc" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.359251 24155 raft_consensus.cc:2454] T 3483d66d5e0b40928016b55c481fdfbc P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 0.
I20240503 02:11:29.359345 24482 tablet_bootstrap.cc:492] T 42b7394682e342fd93c46e68e8b549bd P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.359807 24351 leader_election.cc:302] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters: 844d6da45edb49e19a8c621b067cc515
I20240503 02:11:29.362438 24483 raft_consensus.cc:504] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.362979 24482 tablet_bootstrap.cc:654] T 42b7394682e342fd93c46e68e8b549bd P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.364688 24285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed83724e7f241a3ae68193b1e168b4c" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515"
W20240503 02:11:29.365201 24350 leader_election.cc:341] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.365633 24480 tablet_bootstrap.cc:492] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.365800 24480 ts_tablet_manager.cc:1367] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.015s
I20240503 02:11:29.365708 24483 leader_election.cc:288] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 election: Requested vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079)
I20240503 02:11:29.366261 24483 raft_consensus.cc:2781] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.366626 24483 raft_consensus.cc:482] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.366801 24480 raft_consensus.cc:348] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.367079 24480 raft_consensus.cc:374] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.367177 24480 raft_consensus.cc:729] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.367266 24478 tablet_bootstrap.cc:492] T 7317ad2eec8d42f4bd620fc62d904325 P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.367370 24483 raft_consensus.cc:3037] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.367440 24478 ts_tablet_manager.cc:1367] T 7317ad2eec8d42f4bd620fc62d904325 P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.008s
I20240503 02:11:29.367534 24492 raft_consensus.cc:482] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.367633 24480 consensus_queue.cc:260] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.367749 24492 raft_consensus.cc:504] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.368274 24480 ts_tablet_manager.cc:1398] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:29.368253 24478 raft_consensus.cc:348] T 7317ad2eec8d42f4bd620fc62d904325 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.368414 24478 raft_consensus.cc:374] T 7317ad2eec8d42f4bd620fc62d904325 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.368489 24478 raft_consensus.cc:729] T 7317ad2eec8d42f4bd620fc62d904325 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.368528 24480 tablet_bootstrap.cc:492] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.368687 24478 consensus_queue.cc:260] T 7317ad2eec8d42f4bd620fc62d904325 P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.369177 24478 ts_tablet_manager.cc:1398] T 7317ad2eec8d42f4bd620fc62d904325 P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:11:29.369423 24478 tablet_bootstrap.cc:492] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.369493 24492 leader_election.cc:288] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.370785 24478 tablet_bootstrap.cc:654] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.373477 24482 tablet_bootstrap.cc:492] T 42b7394682e342fd93c46e68e8b549bd P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.374162 24482 ts_tablet_manager.cc:1367] T 42b7394682e342fd93c46e68e8b549bd P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240503 02:11:29.375433 24482 raft_consensus.cc:348] T 42b7394682e342fd93c46e68e8b549bd P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } }
I20240503 02:11:29.375469 24483 raft_consensus.cc:504] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.376372 24480 tablet_bootstrap.cc:654] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.376789 24482 raft_consensus.cc:374] T 42b7394682e342fd93c46e68e8b549bd P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.377060 24482 raft_consensus.cc:729] T 42b7394682e342fd93c46e68e8b549bd P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.377516 24482 consensus_queue.cc:260] T 42b7394682e342fd93c46e68e8b549bd P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } }
I20240503 02:11:29.378872 24482 ts_tablet_manager.cc:1398] T 42b7394682e342fd93c46e68e8b549bd P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:11:29.379791 24482 tablet_bootstrap.cc:492] T 3ed83724e7f241a3ae68193b1e168b4c P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.379873 24285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3483d66d5e0b40928016b55c481fdfbc" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515"
W20240503 02:11:29.380946 24350 leader_election.cc:341] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.382164 24482 tablet_bootstrap.cc:654] T 3ed83724e7f241a3ae68193b1e168b4c P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.388653 24482 tablet_bootstrap.cc:492] T 3ed83724e7f241a3ae68193b1e168b4c P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.388818 24482 ts_tablet_manager.cc:1367] T 3ed83724e7f241a3ae68193b1e168b4c P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 02:11:29.389673 24482 raft_consensus.cc:348] T 3ed83724e7f241a3ae68193b1e168b4c P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.389912 24482 raft_consensus.cc:374] T 3ed83724e7f241a3ae68193b1e168b4c P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.390012 24482 raft_consensus.cc:729] T 3ed83724e7f241a3ae68193b1e168b4c P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.390254 24482 consensus_queue.cc:260] T 3ed83724e7f241a3ae68193b1e168b4c P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.393326 24482 ts_tablet_manager.cc:1398] T 3ed83724e7f241a3ae68193b1e168b4c P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:11:29.393659 24482 tablet_bootstrap.cc:492] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.396214 24482 tablet_bootstrap.cc:654] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.399147 24483 leader_election.cc:288] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 election: Requested vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841)
I20240503 02:11:29.399468 24483 raft_consensus.cc:482] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.399683 24483 raft_consensus.cc:504] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.400092 24480 tablet_bootstrap.cc:492] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.400281 24480 ts_tablet_manager.cc:1367] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.005s
I20240503 02:11:29.400978 24285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd69f29c6854595845edfbc98867aa5" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
I20240503 02:11:29.401070 24480 raft_consensus.cc:348] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.401296 24285 raft_consensus.cc:2454] T 6cd69f29c6854595845edfbc98867aa5 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 0.
I20240503 02:11:29.401360 24480 raft_consensus.cc:374] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.401523 24480 raft_consensus.cc:729] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.401762 24480 consensus_queue.cc:260] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.401895 24350 leader_election.cc:302] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, 844d6da45edb49e19a8c621b067cc515; no voters:
I20240503 02:11:29.402462 24480 ts_tablet_manager.cc:1398] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:11:29.402768 24480 tablet_bootstrap.cc:492] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.404429 24480 tablet_bootstrap.cc:654] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.407212 24482 tablet_bootstrap.cc:492] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.407387 24482 ts_tablet_manager.cc:1367] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240503 02:11:29.407742 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c991b40002b14c6ba8aaf6ab920f26d4" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.408028 24155 raft_consensus.cc:2454] T c991b40002b14c6ba8aaf6ab920f26d4 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 844d6da45edb49e19a8c621b067cc515 in term 0.
I20240503 02:11:29.408253 24482 raft_consensus.cc:348] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.408516 24482 raft_consensus.cc:374] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.408608 24482 raft_consensus.cc:729] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.408696 24221 leader_election.cc:302] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 844d6da45edb49e19a8c621b067cc515, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.408900 24482 consensus_queue.cc:260] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.409888 24492 raft_consensus.cc:2781] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.410068 24492 raft_consensus.cc:482] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.410182 24492 raft_consensus.cc:3037] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.412336 24482 ts_tablet_manager.cc:1398] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240503 02:11:29.412395 24480 tablet_bootstrap.cc:492] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.412570 24480 ts_tablet_manager.cc:1367] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 02:11:29.412828 24482 tablet_bootstrap.cc:492] T b16a60476daa4740b32781ecf3b1554d P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.413286 24480 raft_consensus.cc:348] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.413542 24480 raft_consensus.cc:374] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.413619 24480 raft_consensus.cc:729] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.413880 24480 consensus_queue.cc:260] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.414353 24482 tablet_bootstrap.cc:654] T b16a60476daa4740b32781ecf3b1554d P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.414505 24480 ts_tablet_manager.cc:1398] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:11:29.414798 24480 tablet_bootstrap.cc:492] T 3173cd7821d6431388828d6e40627791 P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.415002 24492 raft_consensus.cc:504] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.417021 24492 leader_election.cc:288] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 election: Requested vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.417737 24482 tablet_bootstrap.cc:492] T b16a60476daa4740b32781ecf3b1554d P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.417863 24482 ts_tablet_manager.cc:1367] T b16a60476daa4740b32781ecf3b1554d P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 02:11:29.418709 24480 tablet_bootstrap.cc:654] T 3173cd7821d6431388828d6e40627791 P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.419216 24482 raft_consensus.cc:348] T b16a60476daa4740b32781ecf3b1554d P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.419400 24482 raft_consensus.cc:374] T b16a60476daa4740b32781ecf3b1554d P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.419498 24482 raft_consensus.cc:729] T b16a60476daa4740b32781ecf3b1554d P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.419709 24482 consensus_queue.cc:260] T b16a60476daa4740b32781ecf3b1554d P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.420384 24482 ts_tablet_manager.cc:1398] T b16a60476daa4740b32781ecf3b1554d P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:29.420404 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c991b40002b14c6ba8aaf6ab920f26d4" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369"
I20240503 02:11:29.420640 24155 raft_consensus.cc:3037] T c991b40002b14c6ba8aaf6ab920f26d4 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.420671 24482 tablet_bootstrap.cc:492] T 3173cd7821d6431388828d6e40627791 P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.421085 24493 raft_consensus.cc:482] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.421219 24493 raft_consensus.cc:504] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.421890 24493 leader_election.cc:288] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.422528 24482 tablet_bootstrap.cc:654] T 3173cd7821d6431388828d6e40627791 P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.422468 24415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5e5a2517bd48b7a58cee8cb27facd2" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827" is_pre_election: true
I20240503 02:11:29.422684 24415 raft_consensus.cc:2454] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd6be00943a94f71aec0a4dd88593369 in term 0.
I20240503 02:11:29.423208 24089 leader_election.cc:302] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.423496 24493 raft_consensus.cc:2781] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.423650 24493 raft_consensus.cc:482] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.423765 24493 raft_consensus.cc:3037] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.427187 24493 raft_consensus.cc:504] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.428032 24155 raft_consensus.cc:2454] T c991b40002b14c6ba8aaf6ab920f26d4 P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 844d6da45edb49e19a8c621b067cc515 in term 1.
I20240503 02:11:29.428822 24493 leader_election.cc:288] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Requested vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.429301 24221 leader_election.cc:302] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 844d6da45edb49e19a8c621b067cc515, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.429319 24415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5e5a2517bd48b7a58cee8cb27facd2" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827"
I20240503 02:11:29.429577 24415 raft_consensus.cc:3037] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.429975 24285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5e5a2517bd48b7a58cee8cb27facd2" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
I20240503 02:11:29.430636 24285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5e5a2517bd48b7a58cee8cb27facd2" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515"
W20240503 02:11:29.431167 24090 leader_election.cc:341] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:11:29.431596 24090 leader_election.cc:341] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.433135 24415 raft_consensus.cc:2454] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd6be00943a94f71aec0a4dd88593369 in term 1.
I20240503 02:11:29.433485 24492 raft_consensus.cc:2781] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.433612 24089 leader_election.cc:302] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters: 844d6da45edb49e19a8c621b067cc515
I20240503 02:11:29.433681 24492 raft_consensus.cc:686] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [term 1 LEADER]: Becoming Leader. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Running, Role: LEADER
I20240503 02:11:29.434216 24493 raft_consensus.cc:2781] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election won for term 1
W20240503 02:11:29.434247 24331 tablet.cc:2369] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:29.434402 24493 raft_consensus.cc:686] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [term 1 LEADER]: Becoming Leader. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Running, Role: LEADER
I20240503 02:11:29.434713 24493 consensus_queue.cc:237] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.436584 24492 consensus_queue.cc:237] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.437132 24480 tablet_bootstrap.cc:492] T 3173cd7821d6431388828d6e40627791 P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.437315 24480 ts_tablet_manager.cc:1367] T 3173cd7821d6431388828d6e40627791 P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.012s
I20240503 02:11:29.438251 24480 raft_consensus.cc:348] T 3173cd7821d6431388828d6e40627791 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.438519 24480 raft_consensus.cc:374] T 3173cd7821d6431388828d6e40627791 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.438612 24480 raft_consensus.cc:729] T 3173cd7821d6431388828d6e40627791 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.438848 24480 consensus_queue.cc:260] T 3173cd7821d6431388828d6e40627791 P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.439553 24480 ts_tablet_manager.cc:1398] T 3173cd7821d6431388828d6e40627791 P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:11:29.439837 24480 tablet_bootstrap.cc:492] T 2e94ed8ccadf4b22bbe16211a7dfab4c P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.441658 24480 tablet_bootstrap.cc:654] T 2e94ed8ccadf4b22bbe16211a7dfab4c P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.444552 24013 catalog_manager.cc:5484] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd6be00943a94f71aec0a4dd88593369 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "bd6be00943a94f71aec0a4dd88593369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:29.446700 24480 tablet_bootstrap.cc:492] T 2e94ed8ccadf4b22bbe16211a7dfab4c P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.446844 24480 ts_tablet_manager.cc:1367] T 2e94ed8ccadf4b22bbe16211a7dfab4c P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 02:11:29.447521 24480 raft_consensus.cc:348] T 2e94ed8ccadf4b22bbe16211a7dfab4c P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.447746 24480 raft_consensus.cc:374] T 2e94ed8ccadf4b22bbe16211a7dfab4c P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.447865 24480 raft_consensus.cc:729] T 2e94ed8ccadf4b22bbe16211a7dfab4c P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.448120 24480 consensus_queue.cc:260] T 2e94ed8ccadf4b22bbe16211a7dfab4c P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.448688 24480 ts_tablet_manager.cc:1398] T 2e94ed8ccadf4b22bbe16211a7dfab4c P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:29.448984 24480 tablet_bootstrap.cc:492] T dbb09cbbb3604a698492e47e3e61cfcb P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.450970 24480 tablet_bootstrap.cc:654] T dbb09cbbb3604a698492e47e3e61cfcb P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.456127 24529 raft_consensus.cc:482] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.456279 24529 raft_consensus.cc:504] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.457249 24525 raft_consensus.cc:2781] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.457249 24529 leader_election.cc:288] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.457415 24525 raft_consensus.cc:482] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.457584 24525 raft_consensus.cc:3037] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.459404 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e94ed8ccadf4b22bbe16211a7dfab4c" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.459874 24155 raft_consensus.cc:2454] T 2e94ed8ccadf4b22bbe16211a7dfab4c P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 844d6da45edb49e19a8c621b067cc515 in term 0.
I20240503 02:11:29.460517 24221 leader_election.cc:302] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 844d6da45edb49e19a8c621b067cc515, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.460273 24013 catalog_manager.cc:5484] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 reported cstate change: term changed from 0 to 1, leader changed from <none> to 844d6da45edb49e19a8c621b067cc515 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "844d6da45edb49e19a8c621b067cc515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:29.460947 24529 raft_consensus.cc:2781] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.461062 24529 raft_consensus.cc:482] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.461175 24529 raft_consensus.cc:3037] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.461187 24483 leader_election.cc:288] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841)
I20240503 02:11:29.461634 24483 raft_consensus.cc:482] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.461898 24480 tablet_bootstrap.cc:492] T dbb09cbbb3604a698492e47e3e61cfcb P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.461822 24483 raft_consensus.cc:504] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.462029 24480 ts_tablet_manager.cc:1367] T dbb09cbbb3604a698492e47e3e61cfcb P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20240503 02:11:29.462698 24480 raft_consensus.cc:348] T dbb09cbbb3604a698492e47e3e61cfcb P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.462906 24480 raft_consensus.cc:374] T dbb09cbbb3604a698492e47e3e61cfcb P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.463002 24480 raft_consensus.cc:729] T dbb09cbbb3604a698492e47e3e61cfcb P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.463017 24285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c991b40002b14c6ba8aaf6ab920f26d4" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
I20240503 02:11:29.463236 24480 consensus_queue.cc:260] T dbb09cbbb3604a698492e47e3e61cfcb P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.463850 24529 raft_consensus.cc:504] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.464064 24480 ts_tablet_manager.cc:1398] T dbb09cbbb3604a698492e47e3e61cfcb P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:11:29.464416 24529 leader_election.cc:288] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 election: Requested vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.464581 24480 tablet_bootstrap.cc:492] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.467337 24480 tablet_bootstrap.cc:654] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.468843 24529 raft_consensus.cc:482] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.468997 24529 raft_consensus.cc:504] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.469061 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e94ed8ccadf4b22bbe16211a7dfab4c" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369"
I20240503 02:11:29.469336 24155 raft_consensus.cc:3037] T 2e94ed8ccadf4b22bbe16211a7dfab4c P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.469596 24529 leader_election.cc:288] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.473038 24155 raft_consensus.cc:2454] T 2e94ed8ccadf4b22bbe16211a7dfab4c P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 844d6da45edb49e19a8c621b067cc515 in term 1.
I20240503 02:11:29.474088 24221 leader_election.cc:302] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 844d6da45edb49e19a8c621b067cc515, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.474442 24529 raft_consensus.cc:2781] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.474599 24529 raft_consensus.cc:686] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [term 1 LEADER]: Becoming Leader. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Running, Role: LEADER
I20240503 02:11:29.474601 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fa9034b37c4374b469e3f1e5b9cda1" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.474979 24155 raft_consensus.cc:2454] T b1fa9034b37c4374b469e3f1e5b9cda1 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 844d6da45edb49e19a8c621b067cc515 in term 0.
I20240503 02:11:29.474880 24529 consensus_queue.cc:237] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.475898 24480 tablet_bootstrap.cc:492] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.476084 24480 ts_tablet_manager.cc:1367] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240503 02:11:29.475422 24221 leader_election.cc:302] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 844d6da45edb49e19a8c621b067cc515, bd6be00943a94f71aec0a4dd88593369; no voters:
W20240503 02:11:29.477070 24351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.104.193:38079, user_credentials={real_user=slave}} blocked reactor thread for 117275us
I20240503 02:11:29.478057 24154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed83724e7f241a3ae68193b1e168b4c" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369"
I20240503 02:11:29.478214 24480 raft_consensus.cc:348] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.478407 24154 raft_consensus.cc:3037] T 3ed83724e7f241a3ae68193b1e168b4c P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.478508 24480 raft_consensus.cc:374] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.478606 24480 raft_consensus.cc:729] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.478901 24480 consensus_queue.cc:260] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.479720 24480 ts_tablet_manager.cc:1398] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:11:29.480077 24480 tablet_bootstrap.cc:492] T 6cd69f29c6854595845edfbc98867aa5 P bd6be00943a94f71aec0a4dd88593369: Bootstrap starting.
I20240503 02:11:29.482270 24480 tablet_bootstrap.cc:654] T 6cd69f29c6854595845edfbc98867aa5 P bd6be00943a94f71aec0a4dd88593369: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.482470 24154 raft_consensus.cc:2454] T 3ed83724e7f241a3ae68193b1e168b4c P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 1.
I20240503 02:11:29.484046 24519 raft_consensus.cc:482] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.484299 24519 raft_consensus.cc:504] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.485498 24519 leader_election.cc:288] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841)
I20240503 02:11:29.485723 24492 raft_consensus.cc:2781] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.485857 24492 raft_consensus.cc:482] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.485947 24492 raft_consensus.cc:3037] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.485745 24285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb09cbbb3604a698492e47e3e61cfcb" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
I20240503 02:11:29.486233 24285 raft_consensus.cc:2454] T dbb09cbbb3604a698492e47e3e61cfcb P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 0.
I20240503 02:11:29.486694 24350 leader_election.cc:302] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, 844d6da45edb49e19a8c621b067cc515; no voters:
I20240503 02:11:29.487046 24519 raft_consensus.cc:2781] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.487216 24519 raft_consensus.cc:482] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.487404 24519 raft_consensus.cc:3037] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.490361 24154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd69f29c6854595845edfbc98867aa5" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.491186 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3483d66d5e0b40928016b55c481fdfbc" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369"
I20240503 02:11:29.491396 24155 raft_consensus.cc:3037] T 3483d66d5e0b40928016b55c481fdfbc P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.490411 24519 raft_consensus.cc:504] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.493063 24285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb09cbbb3604a698492e47e3e61cfcb" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515"
I20240503 02:11:29.493259 24285 raft_consensus.cc:3037] T dbb09cbbb3604a698492e47e3e61cfcb P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.494251 24480 tablet_bootstrap.cc:492] T 6cd69f29c6854595845edfbc98867aa5 P bd6be00943a94f71aec0a4dd88593369: No bootstrap required, opened a new log
I20240503 02:11:29.494378 24480 ts_tablet_manager.cc:1367] T 6cd69f29c6854595845edfbc98867aa5 P bd6be00943a94f71aec0a4dd88593369: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.008s
I20240503 02:11:29.494426 24155 raft_consensus.cc:2454] T 3483d66d5e0b40928016b55c481fdfbc P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 1.
I20240503 02:11:29.495285 24480 raft_consensus.cc:348] T 6cd69f29c6854595845edfbc98867aa5 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.495499 24480 raft_consensus.cc:374] T 6cd69f29c6854595845edfbc98867aa5 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.495584 24480 raft_consensus.cc:729] T 6cd69f29c6854595845edfbc98867aa5 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.495826 24480 consensus_queue.cc:260] T 6cd69f29c6854595845edfbc98867aa5 P bd6be00943a94f71aec0a4dd88593369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.496381 24480 ts_tablet_manager.cc:1398] T 6cd69f29c6854595845edfbc98867aa5 P bd6be00943a94f71aec0a4dd88593369: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:29.496379 24285 raft_consensus.cc:2454] T dbb09cbbb3604a698492e47e3e61cfcb P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 1.
I20240503 02:11:29.496948 24483 leader_election.cc:288] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841)
I20240503 02:11:29.497057 24350 leader_election.cc:302] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, 844d6da45edb49e19a8c621b067cc515; no voters:
I20240503 02:11:29.497115 24483 raft_consensus.cc:482] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.497766 24483 raft_consensus.cc:504] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.498086 24541 raft_consensus.cc:2781] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.498428 24483 leader_election.cc:288] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079)
I20240503 02:11:29.498728 24541 raft_consensus.cc:686] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [term 1 LEADER]: Becoming Leader. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Running, Role: LEADER
I20240503 02:11:29.498782 24284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e94ed8ccadf4b22bbe16211a7dfab4c" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
I20240503 02:11:29.499687 24541 consensus_queue.cc:237] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.500682 24519 leader_election.cc:288] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 election: Requested vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841)
I20240503 02:11:29.501005 24525 raft_consensus.cc:504] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.502288 24285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd69f29c6854595845edfbc98867aa5" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515"
I20240503 02:11:29.502367 24493 raft_consensus.cc:482] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.502457 24285 raft_consensus.cc:3037] T 6cd69f29c6854595845edfbc98867aa5 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.502565 24493 raft_consensus.cc:504] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.502841 24154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb09cbbb3604a698492e47e3e61cfcb" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.503017 24154 raft_consensus.cc:2454] T dbb09cbbb3604a698492e47e3e61cfcb P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 0.
I20240503 02:11:29.503317 24493 leader_election.cc:288] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.504105 24154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb09cbbb3604a698492e47e3e61cfcb" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369"
I20240503 02:11:29.504081 24415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16b4870e2d34ba89ca7469266f7857b" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827" is_pre_election: true
I20240503 02:11:29.504287 24154 raft_consensus.cc:3037] T dbb09cbbb3604a698492e47e3e61cfcb P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.504302 24415 raft_consensus.cc:2454] T f16b4870e2d34ba89ca7469266f7857b P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd6be00943a94f71aec0a4dd88593369 in term 0.
I20240503 02:11:29.504413 24541 raft_consensus.cc:482] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.504559 24541 raft_consensus.cc:504] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.504763 24089 leader_election.cc:302] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.505051 24493 raft_consensus.cc:2781] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.505158 24493 raft_consensus.cc:482] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.505331 24493 raft_consensus.cc:3037] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.505478 24284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9f1118b6694b88a82dcbbf34c5d8a5" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
W20240503 02:11:29.506006 24350 leader_election.cc:341] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.506134 24541 leader_election.cc:288] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841)
I20240503 02:11:29.506932 24285 raft_consensus.cc:2454] T 6cd69f29c6854595845edfbc98867aa5 P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 1.
I20240503 02:11:29.507139 24154 raft_consensus.cc:2454] T dbb09cbbb3604a698492e47e3e61cfcb P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 1.
I20240503 02:11:29.507381 24350 leader_election.cc:302] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, 844d6da45edb49e19a8c621b067cc515; no voters:
I20240503 02:11:29.507668 24541 raft_consensus.cc:2781] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.507872 24541 raft_consensus.cc:686] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [term 1 LEADER]: Becoming Leader. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Running, Role: LEADER
I20240503 02:11:29.508158 24541 consensus_queue.cc:237] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.508533 24493 raft_consensus.cc:504] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.508502 24153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd69f29c6854595845edfbc98867aa5" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369"
I20240503 02:11:29.509258 24153 raft_consensus.cc:3037] T 6cd69f29c6854595845edfbc98867aa5 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.509321 24154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e94ed8ccadf4b22bbe16211a7dfab4c" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.509820 24154 raft_consensus.cc:2379] T 2e94ed8ccadf4b22bbe16211a7dfab4c P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b26dff28ae1498e85be69e29aa7a827 in current term 1: Already voted for candidate 844d6da45edb49e19a8c621b067cc515 in this term.
I20240503 02:11:29.510448 24493 leader_election.cc:288] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Requested vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.510994 24493 raft_consensus.cc:482] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.511168 24493 raft_consensus.cc:504] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.511999 24153 raft_consensus.cc:2454] T 6cd69f29c6854595845edfbc98867aa5 P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 1.
I20240503 02:11:29.512284 24493 leader_election.cc:288] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.509021 24351 leader_election.cc:302] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters: 844d6da45edb49e19a8c621b067cc515
I20240503 02:11:29.510283 24415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16b4870e2d34ba89ca7469266f7857b" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827"
I20240503 02:11:29.513199 24415 raft_consensus.cc:3037] T f16b4870e2d34ba89ca7469266f7857b P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.513401 24492 raft_consensus.cc:504] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.514214 24492 leader_election.cc:288] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 election: Requested vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.515025 24415 raft_consensus.cc:2454] T f16b4870e2d34ba89ca7469266f7857b P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd6be00943a94f71aec0a4dd88593369 in term 1.
I20240503 02:11:29.512391 24414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9f1118b6694b88a82dcbbf34c5d8a5" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827" is_pre_election: true
I20240503 02:11:29.515321 24414 raft_consensus.cc:2454] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd6be00943a94f71aec0a4dd88593369 in term 0.
I20240503 02:11:29.515487 24153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fa9034b37c4374b469e3f1e5b9cda1" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369"
I20240503 02:11:29.515687 24153 raft_consensus.cc:3037] T b1fa9034b37c4374b469e3f1e5b9cda1 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.515692 24285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16b4870e2d34ba89ca7469266f7857b" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
I20240503 02:11:29.515821 24089 leader_election.cc:302] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.516094 24089 leader_election.cc:302] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.516084 24284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16b4870e2d34ba89ca7469266f7857b" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515"
W20240503 02:11:29.516264 24090 leader_election.cc:341] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.516417 24485 raft_consensus.cc:2781] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.516536 24493 raft_consensus.cc:2781] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.516626 24493 raft_consensus.cc:482] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.516738 24493 raft_consensus.cc:3037] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:11:29.516784 24090 leader_election.cc:341] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.516727 24283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9f1118b6694b88a82dcbbf34c5d8a5" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
W20240503 02:11:29.517287 24090 leader_election.cc:341] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.518007 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c991b40002b14c6ba8aaf6ab920f26d4" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.518328 24155 raft_consensus.cc:2379] T c991b40002b14c6ba8aaf6ab920f26d4 P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b26dff28ae1498e85be69e29aa7a827 in current term 1: Already voted for candidate 844d6da45edb49e19a8c621b067cc515 in this term.
I20240503 02:11:29.519358 24153 raft_consensus.cc:2454] T b1fa9034b37c4374b469e3f1e5b9cda1 P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 844d6da45edb49e19a8c621b067cc515 in term 1.
I20240503 02:11:29.519976 24221 leader_election.cc:302] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 844d6da45edb49e19a8c621b067cc515, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.520009 24493 raft_consensus.cc:504] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.520071 24519 raft_consensus.cc:2781] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.520231 24519 raft_consensus.cc:686] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [term 1 LEADER]: Becoming Leader. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Running, Role: LEADER
W20240503 02:11:29.520280 24351 leader_election.cc:341] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.520591 24351 leader_election.cc:302] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters: 844d6da45edb49e19a8c621b067cc515
I20240503 02:11:29.520619 24519 consensus_queue.cc:237] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.520944 24482 tablet_bootstrap.cc:492] T 3173cd7821d6431388828d6e40627791 P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.521318 24351 leader_election.cc:302] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827; no voters: 844d6da45edb49e19a8c621b067cc515, bd6be00943a94f71aec0a4dd88593369
I20240503 02:11:29.521375 24414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9f1118b6694b88a82dcbbf34c5d8a5" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827"
I20240503 02:11:29.521394 24482 ts_tablet_manager.cc:1367] T 3173cd7821d6431388828d6e40627791 P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.101s user 0.039s sys 0.027s
I20240503 02:11:29.521426 24283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9f1118b6694b88a82dcbbf34c5d8a5" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515"
I20240503 02:11:29.521540 24414 raft_consensus.cc:3037] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.522183 24482 raft_consensus.cc:348] T 3173cd7821d6431388828d6e40627791 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.522408 24482 raft_consensus.cc:374] T 3173cd7821d6431388828d6e40627791 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.522488 24482 raft_consensus.cc:729] T 3173cd7821d6431388828d6e40627791 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.522737 24482 consensus_queue.cc:260] T 3173cd7821d6431388828d6e40627791 P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.523327 24482 ts_tablet_manager.cc:1398] T 3173cd7821d6431388828d6e40627791 P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:11:29.523663 24482 tablet_bootstrap.cc:492] T 3483d66d5e0b40928016b55c481fdfbc P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.523932 24539 raft_consensus.cc:2781] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.525102 24414 raft_consensus.cc:2454] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd6be00943a94f71aec0a4dd88593369 in term 1.
I20240503 02:11:29.525610 24089 leader_election.cc:302] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters:
W20240503 02:11:29.526115 24090 leader_election.cc:341] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.527455 24525 leader_election.cc:288] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 election: Requested vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841)
I20240503 02:11:29.527647 24525 raft_consensus.cc:3037] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.520877 24483 raft_consensus.cc:2781] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.528606 24483 raft_consensus.cc:686] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [term 1 LEADER]: Becoming Leader. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Running, Role: LEADER
I20240503 02:11:29.529204 24483 consensus_queue.cc:237] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.530812 24525 raft_consensus.cc:2726] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:11:29.532125 24013 catalog_manager.cc:5484] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 reported cstate change: term changed from 0 to 1, leader changed from <none> to 844d6da45edb49e19a8c621b067cc515 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "844d6da45edb49e19a8c621b067cc515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:29.538401 24493 leader_election.cc:288] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Requested vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.539242 24543 raft_consensus.cc:2781] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.539474 24543 raft_consensus.cc:686] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [term 1 LEADER]: Becoming Leader. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Running, Role: LEADER
I20240503 02:11:29.539949 24478 tablet_bootstrap.cc:492] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.539726 24485 raft_consensus.cc:686] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [term 1 LEADER]: Becoming Leader. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Running, Role: LEADER
I20240503 02:11:29.540159 24478 ts_tablet_manager.cc:1367] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.171s user 0.054s sys 0.038s
I20240503 02:11:29.540417 24485 consensus_queue.cc:237] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.540917 24478 raft_consensus.cc:348] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.541131 24478 raft_consensus.cc:374] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.541218 24478 raft_consensus.cc:729] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.541467 24478 consensus_queue.cc:260] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.538596 24539 raft_consensus.cc:686] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [term 1 LEADER]: Becoming Leader. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Running, Role: LEADER
I20240503 02:11:29.541991 24478 ts_tablet_manager.cc:1398] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:29.542016 24539 consensus_queue.cc:237] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.542304 24478 tablet_bootstrap.cc:492] T b1fa9034b37c4374b469e3f1e5b9cda1 P 7b26dff28ae1498e85be69e29aa7a827: Bootstrap starting.
I20240503 02:11:29.542146 24543 consensus_queue.cc:237] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.544000 24478 tablet_bootstrap.cc:654] T b1fa9034b37c4374b469e3f1e5b9cda1 P 7b26dff28ae1498e85be69e29aa7a827: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.546988 24482 tablet_bootstrap.cc:654] T 3483d66d5e0b40928016b55c481fdfbc P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.547258 24351 leader_election.cc:302] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827; no voters: 844d6da45edb49e19a8c621b067cc515, bd6be00943a94f71aec0a4dd88593369
I20240503 02:11:29.548960 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9f1118b6694b88a82dcbbf34c5d8a5" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.549676 24493 raft_consensus.cc:482] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.549952 24493 raft_consensus.cc:504] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } }
I20240503 02:11:29.551945 24493 leader_election.cc:288] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.552565 24544 raft_consensus.cc:3037] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.552707 24013 catalog_manager.cc:5484] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 reported cstate change: term changed from 0 to 1, leader changed from <none> to 844d6da45edb49e19a8c621b067cc515 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "844d6da45edb49e19a8c621b067cc515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:29.553673 24012 catalog_manager.cc:5484] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd6be00943a94f71aec0a4dd88593369 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "bd6be00943a94f71aec0a4dd88593369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:29.555388 24544 raft_consensus.cc:2726] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:11:29.555511 24414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b7394682e342fd93c46e68e8b549bd" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827" is_pre_election: true
I20240503 02:11:29.555737 24414 raft_consensus.cc:2454] T 42b7394682e342fd93c46e68e8b549bd P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd6be00943a94f71aec0a4dd88593369 in term 0.
I20240503 02:11:29.558456 24089 leader_election.cc:302] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.558537 24415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c991b40002b14c6ba8aaf6ab920f26d4" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827"
I20240503 02:11:29.559199 24485 raft_consensus.cc:2781] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.559316 24485 raft_consensus.cc:482] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.559439 24485 raft_consensus.cc:3037] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.560077 24283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b7394682e342fd93c46e68e8b549bd" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
I20240503 02:11:29.560334 24283 raft_consensus.cc:2454] T 42b7394682e342fd93c46e68e8b549bd P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd6be00943a94f71aec0a4dd88593369 in term 0.
I20240503 02:11:29.562476 24482 tablet_bootstrap.cc:492] T 3483d66d5e0b40928016b55c481fdfbc P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.558426 24413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e94ed8ccadf4b22bbe16211a7dfab4c" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827" is_pre_election: true
I20240503 02:11:29.563019 24482 ts_tablet_manager.cc:1367] T 3483d66d5e0b40928016b55c481fdfbc P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20240503 02:11:29.564071 24482 raft_consensus.cc:348] T 3483d66d5e0b40928016b55c481fdfbc P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.564426 24482 raft_consensus.cc:374] T 3483d66d5e0b40928016b55c481fdfbc P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.564576 24482 raft_consensus.cc:729] T 3483d66d5e0b40928016b55c481fdfbc P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.564936 24482 consensus_queue.cc:260] T 3483d66d5e0b40928016b55c481fdfbc P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.565940 24482 ts_tablet_manager.cc:1398] T 3483d66d5e0b40928016b55c481fdfbc P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:29.565418 24013 catalog_manager.cc:5484] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195). New cstate: current_term: 1 leader_uuid: "7b26dff28ae1498e85be69e29aa7a827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:29.566938 24013 catalog_manager.cc:5484] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195). New cstate: current_term: 1 leader_uuid: "7b26dff28ae1498e85be69e29aa7a827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:29.568497 24539 raft_consensus.cc:482] T 3483d66d5e0b40928016b55c481fdfbc P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.568758 24539 raft_consensus.cc:504] T 3483d66d5e0b40928016b55c481fdfbc P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.570032 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3483d66d5e0b40928016b55c481fdfbc" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.570363 24155 raft_consensus.cc:2379] T 3483d66d5e0b40928016b55c481fdfbc P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 844d6da45edb49e19a8c621b067cc515 in current term 1: Already voted for candidate 7b26dff28ae1498e85be69e29aa7a827 in this term.
I20240503 02:11:29.571249 24485 raft_consensus.cc:504] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } }
I20240503 02:11:29.571563 24413 raft_consensus.cc:2454] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 844d6da45edb49e19a8c621b067cc515 in term 1.
I20240503 02:11:29.571988 24411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fa9034b37c4374b469e3f1e5b9cda1" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827" is_pre_election: true
I20240503 02:11:29.572672 24283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b7394682e342fd93c46e68e8b549bd" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515"
I20240503 02:11:29.572886 24283 raft_consensus.cc:3037] T 42b7394682e342fd93c46e68e8b549bd P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.572796 24410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fa9034b37c4374b469e3f1e5b9cda1" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827"
I20240503 02:11:29.575067 24539 leader_election.cc:288] T 3483d66d5e0b40928016b55c481fdfbc P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.577337 24283 raft_consensus.cc:2454] T 42b7394682e342fd93c46e68e8b549bd P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd6be00943a94f71aec0a4dd88593369 in term 1.
I20240503 02:11:29.577705 24351 leader_election.cc:302] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827; no voters: 844d6da45edb49e19a8c621b067cc515, bd6be00943a94f71aec0a4dd88593369
I20240503 02:11:29.578142 24544 raft_consensus.cc:2726] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:11:29.578931 24090 leader_election.cc:302] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 844d6da45edb49e19a8c621b067cc515, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.579278 24548 raft_consensus.cc:482] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 02:11:29.579541 24219 leader_election.cc:341] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.579469 24548 raft_consensus.cc:504] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
W20240503 02:11:29.580184 24219 leader_election.cc:341] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.580437 24548 leader_election.cc:288] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.580657 24548 raft_consensus.cc:2781] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.580834 24548 raft_consensus.cc:686] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [term 1 LEADER]: Becoming Leader. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Running, Role: LEADER
I20240503 02:11:29.581187 24548 consensus_queue.cc:237] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } }
I20240503 02:11:29.582557 24410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3483d66d5e0b40928016b55c481fdfbc" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827" is_pre_election: true
I20240503 02:11:29.583492 24012 catalog_manager.cc:5484] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195). New cstate: current_term: 1 leader_uuid: "7b26dff28ae1498e85be69e29aa7a827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:29.583984 24482 tablet_bootstrap.cc:492] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.584436 24012 catalog_manager.cc:5484] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195). New cstate: current_term: 1 leader_uuid: "7b26dff28ae1498e85be69e29aa7a827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:29.585839 24283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7317ad2eec8d42f4bd620fc62d904325" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
I20240503 02:11:29.586644 24219 leader_election.cc:302] T 3483d66d5e0b40928016b55c481fdfbc P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 844d6da45edb49e19a8c621b067cc515; no voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369
W20240503 02:11:29.588754 24090 leader_election.cc:341] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.589427 24410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7317ad2eec8d42f4bd620fc62d904325" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827" is_pre_election: true
I20240503 02:11:29.589609 24410 raft_consensus.cc:2454] T 7317ad2eec8d42f4bd620fc62d904325 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd6be00943a94f71aec0a4dd88593369 in term 0.
I20240503 02:11:29.589972 24539 raft_consensus.cc:3037] T 3483d66d5e0b40928016b55c481fdfbc P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.590035 24089 leader_election.cc:302] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters: 844d6da45edb49e19a8c621b067cc515
I20240503 02:11:29.590617 24543 raft_consensus.cc:2781] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.587895 24414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c991b40002b14c6ba8aaf6ab920f26d4" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827" is_pre_election: true
I20240503 02:11:29.590811 24543 raft_consensus.cc:482] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.590902 24543 raft_consensus.cc:3037] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.591444 24482 tablet_bootstrap.cc:654] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.592007 24013 catalog_manager.cc:5484] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd6be00943a94f71aec0a4dd88593369 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "bd6be00943a94f71aec0a4dd88593369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:29.588761 24485 leader_election.cc:288] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Requested vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.593477 24543 raft_consensus.cc:504] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.594017 24543 leader_election.cc:288] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Requested vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.594636 24283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7317ad2eec8d42f4bd620fc62d904325" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515"
W20240503 02:11:29.595043 24090 leader_election.cc:341] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:29.589263 24412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e94ed8ccadf4b22bbe16211a7dfab4c" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827"
I20240503 02:11:29.595654 24411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b7394682e342fd93c46e68e8b549bd" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827"
I20240503 02:11:29.597038 24013 catalog_manager.cc:5484] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd6be00943a94f71aec0a4dd88593369 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "bd6be00943a94f71aec0a4dd88593369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:29.599907 24411 raft_consensus.cc:3037] T 42b7394682e342fd93c46e68e8b549bd P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.600004 24414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7317ad2eec8d42f4bd620fc62d904325" candidate_uuid: "bd6be00943a94f71aec0a4dd88593369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827"
I20240503 02:11:29.600209 24414 raft_consensus.cc:3037] T 7317ad2eec8d42f4bd620fc62d904325 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.602144 24539 raft_consensus.cc:2726] T 3483d66d5e0b40928016b55c481fdfbc P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:11:29.602438 24478 tablet_bootstrap.cc:492] T b1fa9034b37c4374b469e3f1e5b9cda1 P 7b26dff28ae1498e85be69e29aa7a827: No bootstrap required, opened a new log
I20240503 02:11:29.602591 24478 ts_tablet_manager.cc:1367] T b1fa9034b37c4374b469e3f1e5b9cda1 P 7b26dff28ae1498e85be69e29aa7a827: Time spent bootstrapping tablet: real 0.060s user 0.024s sys 0.022s
I20240503 02:11:29.603335 24415 raft_consensus.cc:2454] T c991b40002b14c6ba8aaf6ab920f26d4 P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 844d6da45edb49e19a8c621b067cc515 in term 1.
I20240503 02:11:29.603636 24412 raft_consensus.cc:2454] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 844d6da45edb49e19a8c621b067cc515 in term 1.
I20240503 02:11:29.604171 24482 tablet_bootstrap.cc:492] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.604092 24414 raft_consensus.cc:2454] T 7317ad2eec8d42f4bd620fc62d904325 P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd6be00943a94f71aec0a4dd88593369 in term 1.
I20240503 02:11:29.604424 24482 ts_tablet_manager.cc:1367] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.001s
I20240503 02:11:29.604564 24089 leader_election.cc:302] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters: 844d6da45edb49e19a8c621b067cc515
I20240503 02:11:29.604434 24478 raft_consensus.cc:348] T b1fa9034b37c4374b469e3f1e5b9cda1 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.604758 24478 raft_consensus.cc:374] T b1fa9034b37c4374b469e3f1e5b9cda1 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.604861 24478 raft_consensus.cc:729] T b1fa9034b37c4374b469e3f1e5b9cda1 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.605103 24543 raft_consensus.cc:2781] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.605096 24478 consensus_queue.cc:260] T b1fa9034b37c4374b469e3f1e5b9cda1 P 7b26dff28ae1498e85be69e29aa7a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.605652 24478 ts_tablet_manager.cc:1398] T b1fa9034b37c4374b469e3f1e5b9cda1 P 7b26dff28ae1498e85be69e29aa7a827: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:11:29.605669 24482 raft_consensus.cc:348] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.605862 24411 raft_consensus.cc:2454] T 42b7394682e342fd93c46e68e8b549bd P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd6be00943a94f71aec0a4dd88593369 in term 1.
I20240503 02:11:29.605904 24482 raft_consensus.cc:374] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.605922 24543 raft_consensus.cc:686] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [term 1 LEADER]: Becoming Leader. State: Replica: bd6be00943a94f71aec0a4dd88593369, State: Running, Role: LEADER
I20240503 02:11:29.605976 24482 raft_consensus.cc:729] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.606238 24482 consensus_queue.cc:260] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.606413 24543 consensus_queue.cc:237] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.607002 24482 ts_tablet_manager.cc:1398] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:11:29.607319 24482 tablet_bootstrap.cc:492] T 7317ad2eec8d42f4bd620fc62d904325 P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.609437 24012 catalog_manager.cc:5484] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd6be00943a94f71aec0a4dd88593369 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "bd6be00943a94f71aec0a4dd88593369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:29.610265 24482 tablet_bootstrap.cc:654] T 7317ad2eec8d42f4bd620fc62d904325 P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.612686 24482 tablet_bootstrap.cc:492] T 7317ad2eec8d42f4bd620fc62d904325 P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.612792 24482 ts_tablet_manager.cc:1367] T 7317ad2eec8d42f4bd620fc62d904325 P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240503 02:11:29.613271 24482 raft_consensus.cc:348] T 7317ad2eec8d42f4bd620fc62d904325 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.613409 24482 raft_consensus.cc:374] T 7317ad2eec8d42f4bd620fc62d904325 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.613471 24482 raft_consensus.cc:729] T 7317ad2eec8d42f4bd620fc62d904325 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.613689 24482 consensus_queue.cc:260] T 7317ad2eec8d42f4bd620fc62d904325 P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.614121 24482 ts_tablet_manager.cc:1398] T 7317ad2eec8d42f4bd620fc62d904325 P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:11:29.614339 24482 tablet_bootstrap.cc:492] T f16b4870e2d34ba89ca7469266f7857b P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.614779 24544 raft_consensus.cc:482] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.614935 24544 raft_consensus.cc:504] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.615540 24544 leader_election.cc:288] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841)
I20240503 02:11:29.615896 24482 tablet_bootstrap.cc:654] T f16b4870e2d34ba89ca7469266f7857b P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.615912 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16a60476daa4740b32781ecf3b1554d" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.616133 24155 raft_consensus.cc:2454] T b16a60476daa4740b32781ecf3b1554d P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 0.
I20240503 02:11:29.616194 24283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16a60476daa4740b32781ecf3b1554d" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
I20240503 02:11:29.616427 24283 raft_consensus.cc:2454] T b16a60476daa4740b32781ecf3b1554d P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 0.
I20240503 02:11:29.616477 24351 leader_election.cc:302] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.616912 24544 raft_consensus.cc:2781] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.617013 24544 raft_consensus.cc:482] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.617125 24544 raft_consensus.cc:3037] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.618031 24482 tablet_bootstrap.cc:492] T f16b4870e2d34ba89ca7469266f7857b P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.618153 24482 ts_tablet_manager.cc:1367] T f16b4870e2d34ba89ca7469266f7857b P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:11:29.618611 24482 raft_consensus.cc:348] T f16b4870e2d34ba89ca7469266f7857b P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.618757 24482 raft_consensus.cc:374] T f16b4870e2d34ba89ca7469266f7857b P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.618806 24482 raft_consensus.cc:729] T f16b4870e2d34ba89ca7469266f7857b P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.618994 24482 consensus_queue.cc:260] T f16b4870e2d34ba89ca7469266f7857b P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.619097 24544 raft_consensus.cc:504] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.619385 24482 ts_tablet_manager.cc:1398] T f16b4870e2d34ba89ca7469266f7857b P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:11:29.619612 24482 tablet_bootstrap.cc:492] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 844d6da45edb49e19a8c621b067cc515: Bootstrap starting.
I20240503 02:11:29.619693 24544 leader_election.cc:288] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 election: Requested vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841)
I20240503 02:11:29.620148 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16a60476daa4740b32781ecf3b1554d" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369"
I20240503 02:11:29.620142 24283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16a60476daa4740b32781ecf3b1554d" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515"
I20240503 02:11:29.620311 24155 raft_consensus.cc:3037] T b16a60476daa4740b32781ecf3b1554d P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.620360 24283 raft_consensus.cc:3037] T b16a60476daa4740b32781ecf3b1554d P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.621665 24482 tablet_bootstrap.cc:654] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 844d6da45edb49e19a8c621b067cc515: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.622129 24155 raft_consensus.cc:2454] T b16a60476daa4740b32781ecf3b1554d P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 1.
I20240503 02:11:29.622583 24351 leader_election.cc:302] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.622859 24544 raft_consensus.cc:2781] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.623003 24544 raft_consensus.cc:686] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [term 1 LEADER]: Becoming Leader. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Running, Role: LEADER
I20240503 02:11:29.623307 24544 consensus_queue.cc:237] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.624017 24283 raft_consensus.cc:2454] T b16a60476daa4740b32781ecf3b1554d P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 1.
I20240503 02:11:29.625873 24482 tablet_bootstrap.cc:492] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 844d6da45edb49e19a8c621b067cc515: No bootstrap required, opened a new log
I20240503 02:11:29.625975 24482 ts_tablet_manager.cc:1367] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 844d6da45edb49e19a8c621b067cc515: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240503 02:11:29.626461 24482 raft_consensus.cc:348] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.626660 24482 raft_consensus.cc:374] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.626725 24482 raft_consensus.cc:729] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.626865 24482 consensus_queue.cc:260] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 844d6da45edb49e19a8c621b067cc515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.627255 24482 ts_tablet_manager.cc:1398] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 844d6da45edb49e19a8c621b067cc515: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:11:29.632550 24012 catalog_manager.cc:5484] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195). New cstate: current_term: 1 leader_uuid: "7b26dff28ae1498e85be69e29aa7a827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:29.637806 24544 raft_consensus.cc:482] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.637930 24544 raft_consensus.cc:504] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.638465 24544 leader_election.cc:288] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079)
I20240503 02:11:29.638808 24283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3173cd7821d6431388828d6e40627791" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
I20240503 02:11:29.638880 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3173cd7821d6431388828d6e40627791" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.638974 24283 raft_consensus.cc:2454] T 3173cd7821d6431388828d6e40627791 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 0.
I20240503 02:11:29.639091 24155 raft_consensus.cc:2454] T 3173cd7821d6431388828d6e40627791 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 0.
I20240503 02:11:29.639321 24350 leader_election.cc:302] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, 844d6da45edb49e19a8c621b067cc515; no voters:
I20240503 02:11:29.639578 24544 raft_consensus.cc:2781] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.639668 24544 raft_consensus.cc:482] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.639730 24544 raft_consensus.cc:3037] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.641384 24544 raft_consensus.cc:504] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.641906 24544 leader_election.cc:288] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 election: Requested vote from peers 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841), bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079)
I20240503 02:11:29.642267 24283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3173cd7821d6431388828d6e40627791" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515"
I20240503 02:11:29.642316 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3173cd7821d6431388828d6e40627791" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369"
I20240503 02:11:29.642457 24283 raft_consensus.cc:3037] T 3173cd7821d6431388828d6e40627791 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.642499 24155 raft_consensus.cc:3037] T 3173cd7821d6431388828d6e40627791 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.644225 24283 raft_consensus.cc:2454] T 3173cd7821d6431388828d6e40627791 P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 1.
I20240503 02:11:29.644291 24155 raft_consensus.cc:2454] T 3173cd7821d6431388828d6e40627791 P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b26dff28ae1498e85be69e29aa7a827 in term 1.
I20240503 02:11:29.644584 24350 leader_election.cc:302] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827, 844d6da45edb49e19a8c621b067cc515; no voters:
I20240503 02:11:29.644852 24544 raft_consensus.cc:2781] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.645002 24544 raft_consensus.cc:686] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [term 1 LEADER]: Becoming Leader. State: Replica: 7b26dff28ae1498e85be69e29aa7a827, State: Running, Role: LEADER
I20240503 02:11:29.645251 24544 consensus_queue.cc:237] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.647861 24012 catalog_manager.cc:5484] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195). New cstate: current_term: 1 leader_uuid: "7b26dff28ae1498e85be69e29aa7a827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:29.691349 24539 raft_consensus.cc:482] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.691501 24539 raft_consensus.cc:504] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.692232 24539 leader_election.cc:288] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.692701 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9b3e7e2bc447e69c138cd3a8c78f29" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.692749 24411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9b3e7e2bc447e69c138cd3a8c78f29" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827" is_pre_election: true
I20240503 02:11:29.692971 24155 raft_consensus.cc:2454] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 844d6da45edb49e19a8c621b067cc515 in term 0.
I20240503 02:11:29.692975 24411 raft_consensus.cc:2454] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 844d6da45edb49e19a8c621b067cc515 in term 0.
I20240503 02:11:29.693396 24221 leader_election.cc:302] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 844d6da45edb49e19a8c621b067cc515, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.693708 24539 raft_consensus.cc:2781] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.693826 24539 raft_consensus.cc:482] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.693926 24539 raft_consensus.cc:3037] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.697296 24539 raft_consensus.cc:504] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.698056 24539 leader_election.cc:288] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 election: Requested vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.698308 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9b3e7e2bc447e69c138cd3a8c78f29" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369"
I20240503 02:11:29.698343 24411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9b3e7e2bc447e69c138cd3a8c78f29" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827"
I20240503 02:11:29.698521 24411 raft_consensus.cc:3037] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.698554 24155 raft_consensus.cc:3037] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P bd6be00943a94f71aec0a4dd88593369 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.701263 24411 raft_consensus.cc:2454] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 844d6da45edb49e19a8c621b067cc515 in term 1.
I20240503 02:11:29.701263 24155 raft_consensus.cc:2454] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 844d6da45edb49e19a8c621b067cc515 in term 1.
I20240503 02:11:29.701637 24221 leader_election.cc:302] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 844d6da45edb49e19a8c621b067cc515, bd6be00943a94f71aec0a4dd88593369; no voters:
I20240503 02:11:29.701954 24539 raft_consensus.cc:2781] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.702121 24539 raft_consensus.cc:686] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [term 1 LEADER]: Becoming Leader. State: Replica: 844d6da45edb49e19a8c621b067cc515, State: Running, Role: LEADER
I20240503 02:11:29.702383 24539 consensus_queue.cc:237] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.705822 24012 catalog_manager.cc:5484] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 reported cstate change: term changed from 0 to 1, leader changed from <none> to 844d6da45edb49e19a8c621b067cc515 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "844d6da45edb49e19a8c621b067cc515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:29.718575 24539 raft_consensus.cc:482] T 7317ad2eec8d42f4bd620fc62d904325 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.718737 24539 raft_consensus.cc:504] T 7317ad2eec8d42f4bd620fc62d904325 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.719239 24539 leader_election.cc:288] T 7317ad2eec8d42f4bd620fc62d904325 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.719630 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7317ad2eec8d42f4bd620fc62d904325" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.719678 24411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7317ad2eec8d42f4bd620fc62d904325" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827" is_pre_election: true
I20240503 02:11:29.719978 24411 raft_consensus.cc:2379] T 7317ad2eec8d42f4bd620fc62d904325 P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 844d6da45edb49e19a8c621b067cc515 in current term 1: Already voted for candidate bd6be00943a94f71aec0a4dd88593369 in this term.
I20240503 02:11:29.720328 24219 leader_election.cc:302] T 7317ad2eec8d42f4bd620fc62d904325 P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 844d6da45edb49e19a8c621b067cc515; no voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369
I20240503 02:11:29.720568 24539 raft_consensus.cc:3037] T 7317ad2eec8d42f4bd620fc62d904325 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.723338 24539 raft_consensus.cc:2726] T 7317ad2eec8d42f4bd620fc62d904325 P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:11:29.738651 24283 raft_consensus.cc:3037] T 4bc668713bbe4907a612df4c29acbe7d P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.742197 24543 consensus_queue.cc:1035] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.748620 24485 consensus_queue.cc:1035] T 4bc668713bbe4907a612df4c29acbe7d P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.791528 24539 raft_consensus.cc:482] T 3ed83724e7f241a3ae68193b1e168b4c P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.791707 24539 raft_consensus.cc:504] T 3ed83724e7f241a3ae68193b1e168b4c P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.792340 24539 leader_election.cc:288] T 3ed83724e7f241a3ae68193b1e168b4c P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:29.792861 24411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed83724e7f241a3ae68193b1e168b4c" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827" is_pre_election: true
I20240503 02:11:29.792878 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed83724e7f241a3ae68193b1e168b4c" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.793087 24155 raft_consensus.cc:2379] T 3ed83724e7f241a3ae68193b1e168b4c P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 844d6da45edb49e19a8c621b067cc515 in current term 1: Already voted for candidate 7b26dff28ae1498e85be69e29aa7a827 in this term.
I20240503 02:11:29.793470 24221 leader_election.cc:302] T 3ed83724e7f241a3ae68193b1e168b4c P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 844d6da45edb49e19a8c621b067cc515; no voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369
I20240503 02:11:29.793704 24539 raft_consensus.cc:3037] T 3ed83724e7f241a3ae68193b1e168b4c P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.795569 24539 raft_consensus.cc:2726] T 3ed83724e7f241a3ae68193b1e168b4c P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:11:29.892089 24539 consensus_queue.cc:1035] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.897413 24539 consensus_queue.cc:1035] T 2e94ed8ccadf4b22bbe16211a7dfab4c P 844d6da45edb49e19a8c621b067cc515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.908116 24485 consensus_queue.cc:1035] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.909679 24544 raft_consensus.cc:482] T b1fa9034b37c4374b469e3f1e5b9cda1 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.909891 24544 raft_consensus.cc:504] T b1fa9034b37c4374b469e3f1e5b9cda1 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:29.910494 24544 leader_election.cc:288] T b1fa9034b37c4374b469e3f1e5b9cda1 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841)
I20240503 02:11:29.911065 24283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fa9034b37c4374b469e3f1e5b9cda1" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844d6da45edb49e19a8c621b067cc515" is_pre_election: true
I20240503 02:11:29.912093 24283 raft_consensus.cc:3037] T 7e5e5a2517bd48b7a58cee8cb27facd2 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.913825 24155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fa9034b37c4374b469e3f1e5b9cda1" candidate_uuid: "7b26dff28ae1498e85be69e29aa7a827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:29.914053 24155 raft_consensus.cc:2379] T b1fa9034b37c4374b469e3f1e5b9cda1 P bd6be00943a94f71aec0a4dd88593369 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b26dff28ae1498e85be69e29aa7a827 in current term 1: Already voted for candidate 844d6da45edb49e19a8c621b067cc515 in this term.
I20240503 02:11:29.914500 24351 leader_election.cc:302] T b1fa9034b37c4374b469e3f1e5b9cda1 P 7b26dff28ae1498e85be69e29aa7a827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b26dff28ae1498e85be69e29aa7a827; no voters: 844d6da45edb49e19a8c621b067cc515, bd6be00943a94f71aec0a4dd88593369
I20240503 02:11:29.914933 24544 raft_consensus.cc:3037] T b1fa9034b37c4374b469e3f1e5b9cda1 P 7b26dff28ae1498e85be69e29aa7a827 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.915189 24543 consensus_queue.cc:1035] T 7e5e5a2517bd48b7a58cee8cb27facd2 P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.917255 24544 raft_consensus.cc:2726] T b1fa9034b37c4374b469e3f1e5b9cda1 P 7b26dff28ae1498e85be69e29aa7a827 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:11:29.930843 24539 consensus_queue.cc:1035] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.935106 24544 consensus_queue.cc:1035] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.935679 24539 consensus_queue.cc:1035] T c991b40002b14c6ba8aaf6ab920f26d4 P 844d6da45edb49e19a8c621b067cc515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.938403 24544 consensus_queue.cc:1035] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.948691 24525 consensus_queue.cc:1035] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.949065 24525 consensus_queue.cc:1035] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.954746 24543 consensus_queue.cc:1035] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.978160 24283 raft_consensus.cc:3037] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.981998 24493 consensus_queue.cc:1035] T 3c9f1118b6694b88a82dcbbf34c5d8a5 P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.990410 24519 consensus_queue.cc:1035] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.994526 24541 consensus_queue.cc:1035] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.002278 24541 consensus_queue.cc:1035] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.006467 24525 consensus_queue.cc:1035] T 3483d66d5e0b40928016b55c481fdfbc P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.012274 24529 consensus_queue.cc:1035] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.024461 24493 consensus_queue.cc:1035] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.027803 24283 raft_consensus.cc:3037] T f16b4870e2d34ba89ca7469266f7857b P 844d6da45edb49e19a8c621b067cc515 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:30.030658 24529 raft_consensus.cc:482] T f16b4870e2d34ba89ca7469266f7857b P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:30.030875 24529 raft_consensus.cc:504] T f16b4870e2d34ba89ca7469266f7857b P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 } } peers { permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 } } peers { permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 } }
I20240503 02:11:30.031010 24543 consensus_queue.cc:1035] T f16b4870e2d34ba89ca7469266f7857b P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.031616 24529 leader_election.cc:288] T f16b4870e2d34ba89ca7469266f7857b P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079), 7b26dff28ae1498e85be69e29aa7a827 (127.23.104.195:33273)
I20240503 02:11:30.032002 24411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16b4870e2d34ba89ca7469266f7857b" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26dff28ae1498e85be69e29aa7a827" is_pre_election: true
I20240503 02:11:30.032150 24153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16b4870e2d34ba89ca7469266f7857b" candidate_uuid: "844d6da45edb49e19a8c621b067cc515" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6be00943a94f71aec0a4dd88593369" is_pre_election: true
I20240503 02:11:30.032815 24221 leader_election.cc:302] T f16b4870e2d34ba89ca7469266f7857b P 844d6da45edb49e19a8c621b067cc515 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 844d6da45edb49e19a8c621b067cc515; no voters: 7b26dff28ae1498e85be69e29aa7a827, bd6be00943a94f71aec0a4dd88593369
I20240503 02:11:30.033288 24547 raft_consensus.cc:2726] T f16b4870e2d34ba89ca7469266f7857b P 844d6da45edb49e19a8c621b067cc515 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:11:30.046406 24547 consensus_queue.cc:1035] T b1fa9034b37c4374b469e3f1e5b9cda1 P 844d6da45edb49e19a8c621b067cc515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.048811 24541 consensus_queue.cc:1035] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.054013 24483 consensus_queue.cc:1035] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.069216 24543 consensus_queue.cc:1035] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.074298 24493 consensus_queue.cc:1035] T 42b7394682e342fd93c46e68e8b549bd P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.093163 24547 consensus_queue.cc:1035] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.097957 24492 consensus_queue.cc:1035] T 0f9b3e7e2bc447e69c138cd3a8c78f29 P 844d6da45edb49e19a8c621b067cc515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.146759 24493 consensus_queue.cc:1035] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b26dff28ae1498e85be69e29aa7a827" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 33273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.151216 24483 consensus_queue.cc:1035] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.153306 24548 consensus_queue.cc:1035] T 7317ad2eec8d42f4bd620fc62d904325 P bd6be00943a94f71aec0a4dd88593369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "844d6da45edb49e19a8c621b067cc515" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.161005 24525 consensus_queue.cc:1035] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6be00943a94f71aec0a4dd88593369" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:30.225522 24569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:30.225929 24569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:30.233811 24569 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
W20240503 02:11:30.662145 24090 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:46841, user_credentials={real_user=slave}} blocked reactor thread for 78105.7us
INSERT report
rows total: 2000
time total: 424.407 ms
time per row: 0.212203 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:11:32.202553 24395 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:32.203202 24124 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:32.205528 24252 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:11:32.373693 24395 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:32.374310 24252 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:11:32.376645 24124 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:32.552858 24395 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:32.560111 24252 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:11:32.560568 24124 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:33.159071 24252 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:11:33.166472 24395 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:33.167225 24124 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:33.330804 24395 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:33.331310 24124 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:33.331399 24252 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:11:33.472398 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 24073
I20240503 02:11:33.479812 24195 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:33.599911 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 24073
W20240503 02:11:33.621098 24351 connection.cc:535] client connection to 127.23.104.193:38079 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:11:33.621140 24221 connection.cc:535] server connection from 127.23.104.193:41505 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:11:33.621393 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 24203
W20240503 02:11:33.621374 24351 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:11:33.623231 24351 consensus_peers.cc:485] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 -> Peer bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079): Couldn't send request to peer bd6be00943a94f71aec0a4dd88593369. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:38079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:33.626843 24325 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:11:33.694532 24351 consensus_peers.cc:485] T 6cd69f29c6854595845edfbc98867aa5 P 7b26dff28ae1498e85be69e29aa7a827 -> Peer bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079): Couldn't send request to peer bd6be00943a94f71aec0a4dd88593369. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:38079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:33.700763 24351 consensus_peers.cc:485] T 3173cd7821d6431388828d6e40627791 P 7b26dff28ae1498e85be69e29aa7a827 -> Peer bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079): Couldn't send request to peer bd6be00943a94f71aec0a4dd88593369. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:38079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:33.732476 24351 consensus_peers.cc:485] T 3ed83724e7f241a3ae68193b1e168b4c P 7b26dff28ae1498e85be69e29aa7a827 -> Peer bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079): Couldn't send request to peer bd6be00943a94f71aec0a4dd88593369. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:38079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:33.754043 24221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.104.193:38079: connect: Connection refused (error 111)
I20240503 02:11:33.754207 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 24203
W20240503 02:11:33.775110 24350 connection.cc:535] client connection to 127.23.104.194:46841 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:11:33.775481 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 24333
W20240503 02:11:33.776187 24350 consensus_peers.cc:485] T b16a60476daa4740b32781ecf3b1554d P 7b26dff28ae1498e85be69e29aa7a827 -> Peer 844d6da45edb49e19a8c621b067cc515 (127.23.104.194:46841): Couldn't send request to peer 844d6da45edb49e19a8c621b067cc515. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.194:46841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:33.781277 24455 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:11:33.781333 24351 consensus_peers.cc:485] T dbb09cbbb3604a698492e47e3e61cfcb P 7b26dff28ae1498e85be69e29aa7a827 -> Peer bd6be00943a94f71aec0a4dd88593369 (127.23.104.193:38079): Couldn't send request to peer bd6be00943a94f71aec0a4dd88593369. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:38079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:33.901875 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 24333
I20240503 02:11:33.922283 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 23983
I20240503 02:11:33.923336 24044 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:34.037190 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 23983
2024-05-03T02:11:34Z chronyd exiting
[ OK ] TableRangeRebalancingTest.Basic (10775 ms)
[----------] 1 test from TableRangeRebalancingTest (10776 ms total)
[----------] 1 test from RebalanceStartSafetyTest
[ RUN ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1
I20240503 02:11:34.084884 23971 test_util.cc:268] Using random seed: 1025666302
I20240503 02:11:34.086666 23971 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:11:34.086722 23971 ts_itest-base.cc:113] --------------
I20240503 02:11:34.086771 23971 ts_itest-base.cc:114] 5 tablet servers
I20240503 02:11:34.086822 23971 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:11:34.086884 23971 ts_itest-base.cc:116] --------------
2024-05-03T02:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:11:34Z Disabled control of system clock
I20240503 02:11:34.108366 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.104.254:42561
--webserver_interface=127.23.104.254
--webserver_port=0
--builtin_ntp_servers=127.23.104.212:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.104.254:42561
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 02:11:34.410805 24840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:34.411103 24840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:34.411245 24840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:34.418128 24840 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:34.418254 24840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:11:34.418326 24840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:34.418416 24840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:11:34.418503 24840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:11:34.426716 24840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.23.104.254:42561
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.104.254:42561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.23.104.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:34.427969 24840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:34.429615 24840 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:34.439960 24845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:34.440692 24846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:34.441316 24848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:34.441949 24840 server_base.cc:1034] running on GCE node
I20240503 02:11:34.442612 24840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:34.443897 24840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:34.445091 24840 hybrid_clock.cc:648] HybridClock initialized: now 1714702294445060 us; error 33 us; skew 500 ppm
I20240503 02:11:34.445389 24840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:34.447417 24840 webserver.cc:460] Webserver started at http://127.23.104.254:39083/ using document root <none> and password file <none>
I20240503 02:11:34.448122 24840 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:34.448240 24840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:34.448545 24840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:34.451036 24840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d6dc9db49c014318b429c430e300366b"
format_stamp: "Formatted at 2024-05-03 02:11:34 on dist-test-slave-38ht"
I20240503 02:11:34.451769 24840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d6dc9db49c014318b429c430e300366b"
format_stamp: "Formatted at 2024-05-03 02:11:34 on dist-test-slave-38ht"
I20240503 02:11:34.456447 24840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 02:11:34.459388 24854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:34.460551 24840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:11:34.460722 24840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d6dc9db49c014318b429c430e300366b"
format_stamp: "Formatted at 2024-05-03 02:11:34 on dist-test-slave-38ht"
I20240503 02:11:34.460897 24840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:34.476956 24840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:34.477663 24840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:34.477914 24840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:34.494235 24840 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.254:42561
I20240503 02:11:34.494264 24905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.254:42561 every 8 connection(s)
I20240503 02:11:34.495460 24840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:11:34.498999 24906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:34.502018 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 24840
I20240503 02:11:34.502193 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:11:34.509905 24906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b: Bootstrap starting.
I20240503 02:11:34.513337 24906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:34.514623 24906 log.cc:824] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:34.517431 24906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b: No bootstrap required, opened a new log
I20240503 02:11:34.523391 24906 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dc9db49c014318b429c430e300366b" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 42561 } }
I20240503 02:11:34.523649 24906 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.523731 24906 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6dc9db49c014318b429c430e300366b, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.524263 24906 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dc9db49c014318b429c430e300366b" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 42561 } }
I20240503 02:11:34.524441 24906 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:34.524569 24906 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:34.524739 24906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.526648 24906 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dc9db49c014318b429c430e300366b" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 42561 } }
I20240503 02:11:34.527076 24906 leader_election.cc:302] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6dc9db49c014318b429c430e300366b; no voters:
I20240503 02:11:34.527640 24906 leader_election.cc:288] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:34.527763 24911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:34.529501 24906 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:34.529928 24911 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [term 1 LEADER]: Becoming Leader. State: Replica: d6dc9db49c014318b429c430e300366b, State: Running, Role: LEADER
I20240503 02:11:34.530400 24911 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dc9db49c014318b429c430e300366b" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 42561 } }
I20240503 02:11:34.533264 24913 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6dc9db49c014318b429c430e300366b. Latest consensus state: current_term: 1 leader_uuid: "d6dc9db49c014318b429c430e300366b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dc9db49c014318b429c430e300366b" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 42561 } } }
I20240503 02:11:34.533250 24912 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6dc9db49c014318b429c430e300366b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dc9db49c014318b429c430e300366b" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 42561 } } }
I20240503 02:11:34.533665 24913 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:34.533663 24912 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:34.535125 24921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:34.538980 24921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:34.546078 24921 catalog_manager.cc:1347] Generated new cluster ID: 6636171b351c4b749382e101608595d6
I20240503 02:11:34.546195 24921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:34.568758 24921 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:34.570344 24921 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:34.584322 24921 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d6dc9db49c014318b429c430e300366b: Generated new TSK 0
I20240503 02:11:34.585131 24921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:34.594969 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.193:0
--local_ip_for_outbound_sockets=127.23.104.193
--webserver_interface=127.23.104.193
--webserver_port=0
--tserver_master_addrs=127.23.104.254:42561
--builtin_ntp_servers=127.23.104.212:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:11:34.906639 24930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:34.906904 24930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:34.906989 24930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:34.914121 24930 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:34.914243 24930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:34.914352 24930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.193
I20240503 02:11:34.922928 24930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.23.104.193
--webserver_port=0
--tserver_master_addrs=127.23.104.254:42561
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.193
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:34.924135 24930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:34.925755 24930 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:34.937470 24935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:34.938238 24936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:34.938745 24938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:34.939430 24930 server_base.cc:1034] running on GCE node
I20240503 02:11:34.939981 24930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:34.940824 24930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:34.942015 24930 hybrid_clock.cc:648] HybridClock initialized: now 1714702294941976 us; error 32 us; skew 500 ppm
I20240503 02:11:34.942306 24930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:34.944370 24930 webserver.cc:460] Webserver started at http://127.23.104.193:36287/ using document root <none> and password file <none>
I20240503 02:11:34.945032 24930 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:34.945137 24930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:34.945447 24930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:34.947918 24930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "254cb0f53bcf45dab966a5cc02ecb6f8"
format_stamp: "Formatted at 2024-05-03 02:11:34 on dist-test-slave-38ht"
I20240503 02:11:34.948694 24930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "254cb0f53bcf45dab966a5cc02ecb6f8"
format_stamp: "Formatted at 2024-05-03 02:11:34 on dist-test-slave-38ht"
I20240503 02:11:34.953487 24930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:11:34.956499 24944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:34.957724 24930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:11:34.957865 24930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "254cb0f53bcf45dab966a5cc02ecb6f8"
format_stamp: "Formatted at 2024-05-03 02:11:34 on dist-test-slave-38ht"
I20240503 02:11:34.958035 24930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:34.989575 24930 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:34.990412 24930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:34.990659 24930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:34.991572 24930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:34.992935 24930 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:34.993016 24930 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:34.993129 24930 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:34.993211 24930 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:35.023247 24930 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.193:39109
I20240503 02:11:35.023314 25056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.193:39109 every 8 connection(s)
I20240503 02:11:35.024520 24930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:11:35.030815 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 24930
I20240503 02:11:35.031028 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:11:35.034511 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.194:0
--local_ip_for_outbound_sockets=127.23.104.194
--webserver_interface=127.23.104.194
--webserver_port=0
--tserver_master_addrs=127.23.104.254:42561
--builtin_ntp_servers=127.23.104.212:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:35.036943 25057 heartbeater.cc:347] Connected to a master server at 127.23.104.254:42561
I20240503 02:11:35.037254 25057 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:35.038194 25057 heartbeater.cc:511] Master 127.23.104.254:42561 requested a full tablet report, sending...
I20240503 02:11:35.040264 24871 ts_manager.cc:194] Registered new tserver with Master: 254cb0f53bcf45dab966a5cc02ecb6f8 (127.23.104.193:39109)
I20240503 02:11:35.041597 24871 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.193:54975
W20240503 02:11:35.341923 25061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:35.342185 25061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:35.342299 25061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:35.349303 25061 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:35.349411 25061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:35.349606 25061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.194
I20240503 02:11:35.358232 25061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.23.104.194
--webserver_port=0
--tserver_master_addrs=127.23.104.254:42561
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.194
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:35.359371 25061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:35.360942 25061 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:35.372138 25069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:35.372898 25067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:35.373430 25066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:35.374120 25061 server_base.cc:1034] running on GCE node
I20240503 02:11:35.374619 25061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:35.375382 25061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:35.376560 25061 hybrid_clock.cc:648] HybridClock initialized: now 1714702295376520 us; error 38 us; skew 500 ppm
I20240503 02:11:35.376837 25061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:35.384109 25061 webserver.cc:460] Webserver started at http://127.23.104.194:35315/ using document root <none> and password file <none>
I20240503 02:11:35.384799 25061 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:35.384907 25061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:35.385188 25061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:35.389532 25061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "be4d24804e7b4792b39117aa4dfe7ed1"
format_stamp: "Formatted at 2024-05-03 02:11:35 on dist-test-slave-38ht"
I20240503 02:11:35.390221 25061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "be4d24804e7b4792b39117aa4dfe7ed1"
format_stamp: "Formatted at 2024-05-03 02:11:35 on dist-test-slave-38ht"
I20240503 02:11:35.395061 25061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 02:11:35.398087 25075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:35.399256 25061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 02:11:35.399422 25061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "be4d24804e7b4792b39117aa4dfe7ed1"
format_stamp: "Formatted at 2024-05-03 02:11:35 on dist-test-slave-38ht"
I20240503 02:11:35.399583 25061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:35.429215 25061 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:35.429984 25061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:35.430284 25061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:35.431219 25061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:35.432631 25061 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:35.432718 25061 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:35.432821 25061 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:35.432891 25061 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:35.462015 25061 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.194:38941
I20240503 02:11:35.462080 25187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.194:38941 every 8 connection(s)
I20240503 02:11:35.463313 25061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:11:35.473178 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 25061
I20240503 02:11:35.473351 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:11:35.474349 25188 heartbeater.cc:347] Connected to a master server at 127.23.104.254:42561
I20240503 02:11:35.474613 25188 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:35.475466 25188 heartbeater.cc:511] Master 127.23.104.254:42561 requested a full tablet report, sending...
I20240503 02:11:35.476752 24871 ts_manager.cc:194] Registered new tserver with Master: be4d24804e7b4792b39117aa4dfe7ed1 (127.23.104.194:38941)
I20240503 02:11:35.477346 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.195:0
--local_ip_for_outbound_sockets=127.23.104.195
--webserver_interface=127.23.104.195
--webserver_port=0
--tserver_master_addrs=127.23.104.254:42561
--builtin_ntp_servers=127.23.104.212:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:35.477499 24871 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.194:35989
W20240503 02:11:35.784667 25191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:35.784924 25191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:35.785077 25191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:35.792291 25191 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:35.792399 25191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:35.792548 25191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.195
I20240503 02:11:35.801048 25191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.23.104.195
--webserver_port=0
--tserver_master_addrs=127.23.104.254:42561
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.195
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:35.802220 25191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:35.803969 25191 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:35.815261 25196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:35.815888 25197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:35.816440 25199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:35.817135 25191 server_base.cc:1034] running on GCE node
I20240503 02:11:35.817628 25191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:35.818392 25191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:35.819602 25191 hybrid_clock.cc:648] HybridClock initialized: now 1714702295819569 us; error 24 us; skew 500 ppm
I20240503 02:11:35.819953 25191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:35.821837 25191 webserver.cc:460] Webserver started at http://127.23.104.195:41871/ using document root <none> and password file <none>
I20240503 02:11:35.822429 25191 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:35.822526 25191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:35.822813 25191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:35.825060 25191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f785328e7f4440288020e05bf47b5dc1"
format_stamp: "Formatted at 2024-05-03 02:11:35 on dist-test-slave-38ht"
I20240503 02:11:35.825770 25191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f785328e7f4440288020e05bf47b5dc1"
format_stamp: "Formatted at 2024-05-03 02:11:35 on dist-test-slave-38ht"
I20240503 02:11:35.830427 25191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20240503 02:11:35.833422 25205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:35.834535 25191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:11:35.834686 25191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f785328e7f4440288020e05bf47b5dc1"
format_stamp: "Formatted at 2024-05-03 02:11:35 on dist-test-slave-38ht"
I20240503 02:11:35.834861 25191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:35.847752 25191 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:35.848493 25191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:35.848742 25191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:35.849632 25191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:35.850915 25191 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:35.851006 25191 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:35.851109 25191 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:35.851178 25191 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:35.881542 25191 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.195:45637
I20240503 02:11:35.881593 25317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.195:45637 every 8 connection(s)
I20240503 02:11:35.882745 25191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:11:35.892377 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 25191
I20240503 02:11:35.892575 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:11:35.894594 25318 heartbeater.cc:347] Connected to a master server at 127.23.104.254:42561
I20240503 02:11:35.894887 25318 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:35.895502 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.196:0
--local_ip_for_outbound_sockets=127.23.104.196
--webserver_interface=127.23.104.196
--webserver_port=0
--tserver_master_addrs=127.23.104.254:42561
--builtin_ntp_servers=127.23.104.212:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:35.895679 25318 heartbeater.cc:511] Master 127.23.104.254:42561 requested a full tablet report, sending...
I20240503 02:11:35.897048 24871 ts_manager.cc:194] Registered new tserver with Master: f785328e7f4440288020e05bf47b5dc1 (127.23.104.195:45637)
I20240503 02:11:35.898175 24871 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.195:34213
I20240503 02:11:36.044121 25057 heartbeater.cc:503] Master 127.23.104.254:42561 was elected leader, sending a full tablet report...
W20240503 02:11:36.195288 25321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:36.195536 25321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:36.195616 25321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:36.202629 25321 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:36.202739 25321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:36.202877 25321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.196
I20240503 02:11:36.211359 25321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.23.104.196
--webserver_port=0
--tserver_master_addrs=127.23.104.254:42561
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.196
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:36.212642 25321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:36.214246 25321 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:36.226066 25327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:36.226683 25329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:36.227068 25326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:36.227742 25321 server_base.cc:1034] running on GCE node
I20240503 02:11:36.228286 25321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:36.229141 25321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:36.230333 25321 hybrid_clock.cc:648] HybridClock initialized: now 1714702296230292 us; error 37 us; skew 500 ppm
I20240503 02:11:36.230619 25321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:36.232831 25321 webserver.cc:460] Webserver started at http://127.23.104.196:40797/ using document root <none> and password file <none>
I20240503 02:11:36.233487 25321 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:36.233626 25321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:36.233909 25321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:36.236281 25321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ef6ba29d0cec45b8afea25e2647fe48a"
format_stamp: "Formatted at 2024-05-03 02:11:36 on dist-test-slave-38ht"
I20240503 02:11:36.236958 25321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ef6ba29d0cec45b8afea25e2647fe48a"
format_stamp: "Formatted at 2024-05-03 02:11:36 on dist-test-slave-38ht"
I20240503 02:11:36.241667 25321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:11:36.244607 25335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:36.246255 25321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:36.246402 25321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ef6ba29d0cec45b8afea25e2647fe48a"
format_stamp: "Formatted at 2024-05-03 02:11:36 on dist-test-slave-38ht"
I20240503 02:11:36.246596 25321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:36.267645 25321 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:36.268374 25321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:36.268628 25321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:36.269471 25321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:36.270819 25321 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:36.270907 25321 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:36.271010 25321 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:36.271085 25321 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:36.300608 25321 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.196:45233
I20240503 02:11:36.300668 25447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.196:45233 every 8 connection(s)
I20240503 02:11:36.301872 25321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:11:36.311196 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 25321
I20240503 02:11:36.311354 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:11:36.313443 25448 heartbeater.cc:347] Connected to a master server at 127.23.104.254:42561
I20240503 02:11:36.313736 25448 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:36.314442 25448 heartbeater.cc:511] Master 127.23.104.254:42561 requested a full tablet report, sending...
I20240503 02:11:36.314632 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.197:0
--local_ip_for_outbound_sockets=127.23.104.197
--webserver_interface=127.23.104.197
--webserver_port=0
--tserver_master_addrs=127.23.104.254:42561
--builtin_ntp_servers=127.23.104.212:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:36.315779 24870 ts_manager.cc:194] Registered new tserver with Master: ef6ba29d0cec45b8afea25e2647fe48a (127.23.104.196:45233)
I20240503 02:11:36.316973 24870 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.196:51681
I20240503 02:11:36.479714 25188 heartbeater.cc:503] Master 127.23.104.254:42561 was elected leader, sending a full tablet report...
W20240503 02:11:36.621812 25451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:36.622087 25451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:36.622222 25451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:36.629168 25451 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:36.629294 25451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:36.629437 25451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.197
I20240503 02:11:36.637866 25451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.23.104.197
--webserver_port=0
--tserver_master_addrs=127.23.104.254:42561
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.197
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:36.639110 25451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:36.640780 25451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:36.651928 25456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:36.652422 25457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:36.653160 25459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:36.653772 25451 server_base.cc:1034] running on GCE node
I20240503 02:11:36.654305 25451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:36.655144 25451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:36.656355 25451 hybrid_clock.cc:648] HybridClock initialized: now 1714702296656301 us; error 40 us; skew 500 ppm
I20240503 02:11:36.656695 25451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:36.658831 25451 webserver.cc:460] Webserver started at http://127.23.104.197:41705/ using document root <none> and password file <none>
I20240503 02:11:36.659484 25451 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:36.659593 25451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:36.659961 25451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:36.662569 25451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "791c3dbf39a4437aa46c5f913a731ec8"
format_stamp: "Formatted at 2024-05-03 02:11:36 on dist-test-slave-38ht"
I20240503 02:11:36.663275 25451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "791c3dbf39a4437aa46c5f913a731ec8"
format_stamp: "Formatted at 2024-05-03 02:11:36 on dist-test-slave-38ht"
I20240503 02:11:36.668004 25451 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20240503 02:11:36.670964 25465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:36.672128 25451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:11:36.672259 25451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "791c3dbf39a4437aa46c5f913a731ec8"
format_stamp: "Formatted at 2024-05-03 02:11:36 on dist-test-slave-38ht"
I20240503 02:11:36.672421 25451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:36.700237 25451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:36.701030 25451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:36.701287 25451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:36.702265 25451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:36.703706 25451 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:36.703815 25451 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:36.703933 25451 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:36.704001 25451 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:36.734903 25451 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.197:45109
I20240503 02:11:36.734972 25577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.197:45109 every 8 connection(s)
I20240503 02:11:36.736306 25451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:11:36.740226 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 25451
I20240503 02:11:36.740414 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:11:36.748524 25578 heartbeater.cc:347] Connected to a master server at 127.23.104.254:42561
I20240503 02:11:36.748787 25578 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:36.749482 25578 heartbeater.cc:511] Master 127.23.104.254:42561 requested a full tablet report, sending...
I20240503 02:11:36.750698 24870 ts_manager.cc:194] Registered new tserver with Master: 791c3dbf39a4437aa46c5f913a731ec8 (127.23.104.197:45109)
I20240503 02:11:36.751410 24870 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.197:42351
I20240503 02:11:36.755740 23971 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:11:36.770088 24870 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52314:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 02:11:36.804855 24992 tablet_service.cc:1467] Processing CreateTablet for tablet f9f469d51f1242bb9491dce49f5885a2 (DEFAULT_TABLE table=TestTable [id=7c0be39b76944ad1adcd8501652979e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:36.805444 25383 tablet_service.cc:1467] Processing CreateTablet for tablet f9f469d51f1242bb9491dce49f5885a2 (DEFAULT_TABLE table=TestTable [id=7c0be39b76944ad1adcd8501652979e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:36.806248 24992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f469d51f1242bb9491dce49f5885a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:36.806739 25383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f469d51f1242bb9491dce49f5885a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:36.806589 25513 tablet_service.cc:1467] Processing CreateTablet for tablet f9f469d51f1242bb9491dce49f5885a2 (DEFAULT_TABLE table=TestTable [id=7c0be39b76944ad1adcd8501652979e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:36.807837 25513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f469d51f1242bb9491dce49f5885a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:36.817553 25596 tablet_bootstrap.cc:492] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8: Bootstrap starting.
I20240503 02:11:36.820505 25598 tablet_bootstrap.cc:492] T f9f469d51f1242bb9491dce49f5885a2 P 254cb0f53bcf45dab966a5cc02ecb6f8: Bootstrap starting.
I20240503 02:11:36.820771 25596 tablet_bootstrap.cc:654] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:36.822074 25596 log.cc:824] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:36.823872 25598 tablet_bootstrap.cc:654] T f9f469d51f1242bb9491dce49f5885a2 P 254cb0f53bcf45dab966a5cc02ecb6f8: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:36.823426 25597 tablet_bootstrap.cc:492] T f9f469d51f1242bb9491dce49f5885a2 P ef6ba29d0cec45b8afea25e2647fe48a: Bootstrap starting.
I20240503 02:11:36.825081 25598 log.cc:824] T f9f469d51f1242bb9491dce49f5885a2 P 254cb0f53bcf45dab966a5cc02ecb6f8: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:36.825141 25596 tablet_bootstrap.cc:492] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8: No bootstrap required, opened a new log
I20240503 02:11:36.825356 25596 ts_tablet_manager.cc:1367] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 02:11:36.827474 25597 tablet_bootstrap.cc:654] T f9f469d51f1242bb9491dce49f5885a2 P ef6ba29d0cec45b8afea25e2647fe48a: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:36.827754 25598 tablet_bootstrap.cc:492] T f9f469d51f1242bb9491dce49f5885a2 P 254cb0f53bcf45dab966a5cc02ecb6f8: No bootstrap required, opened a new log
I20240503 02:11:36.828048 25598 ts_tablet_manager.cc:1367] T f9f469d51f1242bb9491dce49f5885a2 P 254cb0f53bcf45dab966a5cc02ecb6f8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 02:11:36.828657 25597 log.cc:824] T f9f469d51f1242bb9491dce49f5885a2 P ef6ba29d0cec45b8afea25e2647fe48a: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:36.831382 25597 tablet_bootstrap.cc:492] T f9f469d51f1242bb9491dce49f5885a2 P ef6ba29d0cec45b8afea25e2647fe48a: No bootstrap required, opened a new log
I20240503 02:11:36.831647 25597 ts_tablet_manager.cc:1367] T f9f469d51f1242bb9491dce49f5885a2 P ef6ba29d0cec45b8afea25e2647fe48a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 02:11:36.831974 25596 raft_consensus.cc:348] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254cb0f53bcf45dab966a5cc02ecb6f8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 39109 } } peers { permanent_uuid: "ef6ba29d0cec45b8afea25e2647fe48a" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 45233 } } peers { permanent_uuid: "791c3dbf39a4437aa46c5f913a731ec8" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45109 } }
I20240503 02:11:36.832432 25596 raft_consensus.cc:374] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:36.832537 25596 raft_consensus.cc:729] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791c3dbf39a4437aa46c5f913a731ec8, State: Initialized, Role: FOLLOWER
I20240503 02:11:36.833236 25596 consensus_queue.cc:260] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254cb0f53bcf45dab966a5cc02ecb6f8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 39109 } } peers { permanent_uuid: "ef6ba29d0cec45b8afea25e2647fe48a" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 45233 } } peers { permanent_uuid: "791c3dbf39a4437aa46c5f913a731ec8" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45109 } }
I20240503 02:11:36.834515 25578 heartbeater.cc:503] Master 127.23.104.254:42561 was elected leader, sending a full tablet report...
I20240503 02:11:36.834897 25596 ts_tablet_manager.cc:1398] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8: Time spent starting tablet: real 0.009s user 0.006s sys 0.004s
I20240503 02:11:36.836617 25598 raft_consensus.cc:348] T f9f469d51f1242bb9491dce49f5885a2 P 254cb0f53bcf45dab966a5cc02ecb6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254cb0f53bcf45dab966a5cc02ecb6f8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 39109 } } peers { permanent_uuid: "ef6ba29d0cec45b8afea25e2647fe48a" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 45233 } } peers { permanent_uuid: "791c3dbf39a4437aa46c5f913a731ec8" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45109 } }
I20240503 02:11:36.837026 25598 raft_consensus.cc:374] T f9f469d51f1242bb9491dce49f5885a2 P 254cb0f53bcf45dab966a5cc02ecb6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:36.837128 25598 raft_consensus.cc:729] T f9f469d51f1242bb9491dce49f5885a2 P 254cb0f53bcf45dab966a5cc02ecb6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254cb0f53bcf45dab966a5cc02ecb6f8, State: Initialized, Role: FOLLOWER
I20240503 02:11:36.837795 25598 consensus_queue.cc:260] T f9f469d51f1242bb9491dce49f5885a2 P 254cb0f53bcf45dab966a5cc02ecb6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254cb0f53bcf45dab966a5cc02ecb6f8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 39109 } } peers { permanent_uuid: "ef6ba29d0cec45b8afea25e2647fe48a" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 45233 } } peers { permanent_uuid: "791c3dbf39a4437aa46c5f913a731ec8" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45109 } }
I20240503 02:11:36.842350 25597 raft_consensus.cc:348] T f9f469d51f1242bb9491dce49f5885a2 P ef6ba29d0cec45b8afea25e2647fe48a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254cb0f53bcf45dab966a5cc02ecb6f8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 39109 } } peers { permanent_uuid: "ef6ba29d0cec45b8afea25e2647fe48a" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 45233 } } peers { permanent_uuid: "791c3dbf39a4437aa46c5f913a731ec8" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45109 } }
I20240503 02:11:36.842856 25597 raft_consensus.cc:374] T f9f469d51f1242bb9491dce49f5885a2 P ef6ba29d0cec45b8afea25e2647fe48a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:36.842990 25597 raft_consensus.cc:729] T f9f469d51f1242bb9491dce49f5885a2 P ef6ba29d0cec45b8afea25e2647fe48a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef6ba29d0cec45b8afea25e2647fe48a, State: Initialized, Role: FOLLOWER
I20240503 02:11:36.843623 25597 consensus_queue.cc:260] T f9f469d51f1242bb9491dce49f5885a2 P ef6ba29d0cec45b8afea25e2647fe48a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254cb0f53bcf45dab966a5cc02ecb6f8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 39109 } } peers { permanent_uuid: "ef6ba29d0cec45b8afea25e2647fe48a" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 45233 } } peers { permanent_uuid: "791c3dbf39a4437aa46c5f913a731ec8" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45109 } }
I20240503 02:11:36.845446 25448 heartbeater.cc:503] Master 127.23.104.254:42561 was elected leader, sending a full tablet report...
I20240503 02:11:36.845479 25598 ts_tablet_manager.cc:1398] T f9f469d51f1242bb9491dce49f5885a2 P 254cb0f53bcf45dab966a5cc02ecb6f8: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20240503 02:11:36.846886 25597 ts_tablet_manager.cc:1398] T f9f469d51f1242bb9491dce49f5885a2 P ef6ba29d0cec45b8afea25e2647fe48a: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20240503 02:11:36.876454 25602 raft_consensus.cc:482] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:36.876648 25602 raft_consensus.cc:504] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254cb0f53bcf45dab966a5cc02ecb6f8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 39109 } } peers { permanent_uuid: "ef6ba29d0cec45b8afea25e2647fe48a" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 45233 } } peers { permanent_uuid: "791c3dbf39a4437aa46c5f913a731ec8" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45109 } }
I20240503 02:11:36.878109 25602 leader_election.cc:288] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 254cb0f53bcf45dab966a5cc02ecb6f8 (127.23.104.193:39109), ef6ba29d0cec45b8afea25e2647fe48a (127.23.104.196:45233)
I20240503 02:11:36.885473 25012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f469d51f1242bb9491dce49f5885a2" candidate_uuid: "791c3dbf39a4437aa46c5f913a731ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254cb0f53bcf45dab966a5cc02ecb6f8" is_pre_election: true
I20240503 02:11:36.885934 25012 raft_consensus.cc:2454] T f9f469d51f1242bb9491dce49f5885a2 P 254cb0f53bcf45dab966a5cc02ecb6f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 791c3dbf39a4437aa46c5f913a731ec8 in term 0.
I20240503 02:11:36.886035 25403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f469d51f1242bb9491dce49f5885a2" candidate_uuid: "791c3dbf39a4437aa46c5f913a731ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6ba29d0cec45b8afea25e2647fe48a" is_pre_election: true
I20240503 02:11:36.886430 25403 raft_consensus.cc:2454] T f9f469d51f1242bb9491dce49f5885a2 P ef6ba29d0cec45b8afea25e2647fe48a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 791c3dbf39a4437aa46c5f913a731ec8 in term 0.
I20240503 02:11:36.886467 25469 leader_election.cc:302] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 254cb0f53bcf45dab966a5cc02ecb6f8, 791c3dbf39a4437aa46c5f913a731ec8; no voters:
I20240503 02:11:36.887638 25602 raft_consensus.cc:2781] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:36.887756 25602 raft_consensus.cc:482] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:36.887866 25602 raft_consensus.cc:3037] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:36.889691 25602 raft_consensus.cc:504] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254cb0f53bcf45dab966a5cc02ecb6f8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 39109 } } peers { permanent_uuid: "ef6ba29d0cec45b8afea25e2647fe48a" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 45233 } } peers { permanent_uuid: "791c3dbf39a4437aa46c5f913a731ec8" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45109 } }
I20240503 02:11:36.890141 25602 leader_election.cc:288] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [CANDIDATE]: Term 1 election: Requested vote from peers 254cb0f53bcf45dab966a5cc02ecb6f8 (127.23.104.193:39109), ef6ba29d0cec45b8afea25e2647fe48a (127.23.104.196:45233)
I20240503 02:11:36.890578 25012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f469d51f1242bb9491dce49f5885a2" candidate_uuid: "791c3dbf39a4437aa46c5f913a731ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254cb0f53bcf45dab966a5cc02ecb6f8"
I20240503 02:11:36.890579 25403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f469d51f1242bb9491dce49f5885a2" candidate_uuid: "791c3dbf39a4437aa46c5f913a731ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6ba29d0cec45b8afea25e2647fe48a"
I20240503 02:11:36.890748 25012 raft_consensus.cc:3037] T f9f469d51f1242bb9491dce49f5885a2 P 254cb0f53bcf45dab966a5cc02ecb6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:36.890748 25403 raft_consensus.cc:3037] T f9f469d51f1242bb9491dce49f5885a2 P ef6ba29d0cec45b8afea25e2647fe48a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:36.893661 25403 raft_consensus.cc:2454] T f9f469d51f1242bb9491dce49f5885a2 P ef6ba29d0cec45b8afea25e2647fe48a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 791c3dbf39a4437aa46c5f913a731ec8 in term 1.
I20240503 02:11:36.893649 25012 raft_consensus.cc:2454] T f9f469d51f1242bb9491dce49f5885a2 P 254cb0f53bcf45dab966a5cc02ecb6f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 791c3dbf39a4437aa46c5f913a731ec8 in term 1.
I20240503 02:11:36.894068 25466 leader_election.cc:302] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 791c3dbf39a4437aa46c5f913a731ec8, ef6ba29d0cec45b8afea25e2647fe48a; no voters:
I20240503 02:11:36.894357 25602 raft_consensus.cc:2781] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:36.894768 25602 raft_consensus.cc:686] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [term 1 LEADER]: Becoming Leader. State: Replica: 791c3dbf39a4437aa46c5f913a731ec8, State: Running, Role: LEADER
I20240503 02:11:36.895243 25602 consensus_queue.cc:237] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254cb0f53bcf45dab966a5cc02ecb6f8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 39109 } } peers { permanent_uuid: "ef6ba29d0cec45b8afea25e2647fe48a" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 45233 } } peers { permanent_uuid: "791c3dbf39a4437aa46c5f913a731ec8" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45109 } }
I20240503 02:11:36.899406 24869 catalog_manager.cc:5484] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 791c3dbf39a4437aa46c5f913a731ec8 (127.23.104.197). New cstate: current_term: 1 leader_uuid: "791c3dbf39a4437aa46c5f913a731ec8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254cb0f53bcf45dab966a5cc02ecb6f8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 39109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6ba29d0cec45b8afea25e2647fe48a" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 45233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "791c3dbf39a4437aa46c5f913a731ec8" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45109 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:36.900395 25318 heartbeater.cc:503] Master 127.23.104.254:42561 was elected leader, sending a full tablet report...
I20240503 02:11:36.914990 23971 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:11:36.917269 23971 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 254cb0f53bcf45dab966a5cc02ecb6f8 to finish bootstrapping
I20240503 02:11:36.925338 23971 ts_itest-base.cc:233] Waiting for 0 tablets on tserver be4d24804e7b4792b39117aa4dfe7ed1 to finish bootstrapping
I20240503 02:11:36.933347 23971 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f785328e7f4440288020e05bf47b5dc1 to finish bootstrapping
I20240503 02:11:36.940816 23971 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ef6ba29d0cec45b8afea25e2647fe48a to finish bootstrapping
I20240503 02:11:36.947384 23971 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 791c3dbf39a4437aa46c5f913a731ec8 to finish bootstrapping
W20240503 02:11:36.987870 25579 tablet.cc:2369] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:11:37.027199 25058 tablet.cc:2369] T f9f469d51f1242bb9491dce49f5885a2 P 254cb0f53bcf45dab966a5cc02ecb6f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:11:37.053857 25449 tablet.cc:2369] T f9f469d51f1242bb9491dce49f5885a2 P ef6ba29d0cec45b8afea25e2647fe48a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:37.279614 25602 consensus_queue.cc:1035] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "254cb0f53bcf45dab966a5cc02ecb6f8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 39109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:37.287474 25602 consensus_queue.cc:1035] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6ba29d0cec45b8afea25e2647fe48a" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 45233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:37.412241 25513 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:37.412613 24992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:37.415442 25123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:37.415828 25253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:37.416648 25383 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:37.582257 25123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:37.582674 24992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:37.590608 25383 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:37.594096 25253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:37.598076 25513 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:37.747524 24992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:37.761979 25123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:37.768649 25383 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:37.774648 25513 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:37.808445 25253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.403695 24992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.404807 25383 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.406956 25123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.416694 25513 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:38.417621 25253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.575740 24992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.584350 25513 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:38.585191 25123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.585265 25383 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.592449 25253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.773670 24992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.784515 25383 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.786371 25253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.794898 25513 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:38.796015 25123 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.953773 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 24930
I20240503 02:11:38.955178 25051 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:39.048455 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 24930
I20240503 02:11:39.064363 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 25061
I20240503 02:11:39.065255 25182 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:39.157357 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 25061
I20240503 02:11:39.174993 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 25191
I20240503 02:11:39.175988 25313 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:39.270447 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 25191
I20240503 02:11:39.287832 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 25321
I20240503 02:11:39.288792 25442 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:11:39.320886 25469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.104.193:39109: connect: Connection refused (error 111)
W20240503 02:11:39.324858 25469 consensus_peers.cc:485] T f9f469d51f1242bb9491dce49f5885a2 P 791c3dbf39a4437aa46c5f913a731ec8 -> Peer 254cb0f53bcf45dab966a5cc02ecb6f8 (127.23.104.193:39109): Couldn't send request to peer 254cb0f53bcf45dab966a5cc02ecb6f8. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:39109: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:39.381414 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 25321
W20240503 02:11:39.398746 24855 connection.cc:535] server connection from 127.23.104.196:51681 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:11:39.398846 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 25451
I20240503 02:11:39.399776 25572 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:39.500543 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 25451
I20240503 02:11:39.519361 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 24840
I20240503 02:11:39.520370 24901 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:39.631052 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 24840
2024-05-03T02:11:39Z chronyd exiting
[ OK ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1 (5589 ms)
[----------] 1 test from RebalanceStartSafetyTest (5589 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/7
I20240503 02:11:39.674311 23971 test_util.cc:268] Using random seed: 1031255716
I20240503 02:11:39.676021 23971 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:11:39.676090 23971 ts_itest-base.cc:113] --------------
I20240503 02:11:39.676136 23971 ts_itest-base.cc:114] 7 tablet servers
I20240503 02:11:39.676185 23971 ts_itest-base.cc:115] 5 replicas per TS
I20240503 02:11:39.676226 23971 ts_itest-base.cc:116] --------------
2024-05-03T02:11:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:11:39Z Disabled control of system clock
I20240503 02:11:39.698532 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.104.254:40709
--webserver_interface=127.23.104.254
--webserver_port=0
--builtin_ntp_servers=127.23.104.212:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.104.254:40709
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 02:11:40.020805 25836 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 02:11:40.021083 25836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:40.021147 25836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:40.021242 25836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:40.028071 25836 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:40.028164 25836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:11:40.028259 25836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:40.028321 25836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:11:40.028409 25836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:11:40.036933 25836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.23.104.254:40709
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.104.254:40709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.23.104.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:40.038162 25836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:40.039920 25836 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:40.050657 25841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:40.051338 25842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:40.051924 25844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:40.052587 25836 server_base.cc:1034] running on GCE node
I20240503 02:11:40.053313 25836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:40.054533 25836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:40.055733 25836 hybrid_clock.cc:648] HybridClock initialized: now 1714702300055679 us; error 30 us; skew 500 ppm
I20240503 02:11:40.056063 25836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:40.058151 25836 webserver.cc:460] Webserver started at http://127.23.104.254:39175/ using document root <none> and password file <none>
I20240503 02:11:40.058813 25836 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:40.058939 25836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:40.059243 25836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:40.061717 25836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b8f9a407d2264aa2ba34e0ddd69ece11"
format_stamp: "Formatted at 2024-05-03 02:11:40 on dist-test-slave-38ht"
I20240503 02:11:40.062413 25836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b8f9a407d2264aa2ba34e0ddd69ece11"
format_stamp: "Formatted at 2024-05-03 02:11:40 on dist-test-slave-38ht"
I20240503 02:11:40.068712 25836 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240503 02:11:40.071689 25850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:40.072993 25836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:11:40.073168 25836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b8f9a407d2264aa2ba34e0ddd69ece11"
format_stamp: "Formatted at 2024-05-03 02:11:40 on dist-test-slave-38ht"
I20240503 02:11:40.073339 25836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:40.098448 25836 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:40.099154 25836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:40.099406 25836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:40.116514 25836 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.254:40709
I20240503 02:11:40.116544 25901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.254:40709 every 8 connection(s)
I20240503 02:11:40.117792 25836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:11:40.122004 25902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:40.124534 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 25836
I20240503 02:11:40.124752 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:11:40.133924 25902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11: Bootstrap starting.
I20240503 02:11:40.137616 25902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:40.138931 25902 log.cc:824] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:40.141501 25902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11: No bootstrap required, opened a new log
I20240503 02:11:40.147531 25902 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f9a407d2264aa2ba34e0ddd69ece11" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 40709 } }
I20240503 02:11:40.147855 25902 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:40.147939 25902 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f9a407d2264aa2ba34e0ddd69ece11, State: Initialized, Role: FOLLOWER
I20240503 02:11:40.148456 25902 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f9a407d2264aa2ba34e0ddd69ece11" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 40709 } }
I20240503 02:11:40.148636 25902 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:40.148727 25902 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:40.148842 25902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:40.166170 25902 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f9a407d2264aa2ba34e0ddd69ece11" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 40709 } }
I20240503 02:11:40.166741 25902 leader_election.cc:302] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8f9a407d2264aa2ba34e0ddd69ece11; no voters:
I20240503 02:11:40.167464 25902 leader_election.cc:288] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:40.167608 25907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:40.169121 25907 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [term 1 LEADER]: Becoming Leader. State: Replica: b8f9a407d2264aa2ba34e0ddd69ece11, State: Running, Role: LEADER
I20240503 02:11:40.169633 25907 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f9a407d2264aa2ba34e0ddd69ece11" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 40709 } }
I20240503 02:11:40.169783 25902 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:40.172394 25908 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8f9a407d2264aa2ba34e0ddd69ece11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f9a407d2264aa2ba34e0ddd69ece11" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 40709 } } }
I20240503 02:11:40.172483 25909 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8f9a407d2264aa2ba34e0ddd69ece11. Latest consensus state: current_term: 1 leader_uuid: "b8f9a407d2264aa2ba34e0ddd69ece11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f9a407d2264aa2ba34e0ddd69ece11" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 40709 } } }
I20240503 02:11:40.172784 25909 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:40.172683 25908 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11 [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:40.174607 25915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:40.179857 25915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:40.188896 25915 catalog_manager.cc:1347] Generated new cluster ID: f393b76b30be4338b7564ea7229c1968
I20240503 02:11:40.188992 25915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:40.196228 25915 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:40.197265 25915 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:40.207168 25915 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b8f9a407d2264aa2ba34e0ddd69ece11: Generated new TSK 0
I20240503 02:11:40.207898 25915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:40.214403 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.193:0
--local_ip_for_outbound_sockets=127.23.104.193
--webserver_interface=127.23.104.193
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--builtin_ntp_servers=127.23.104.212:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:11:40.557864 25926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:40.558136 25926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:40.558259 25926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:40.565716 25926 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:40.565825 25926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:40.565954 25926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.193
I20240503 02:11:40.575101 25926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.23.104.193
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.193
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:40.576524 25926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:40.578203 25926 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:40.590474 25931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:40.591099 25934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:40.591850 25932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:40.592473 25926 server_base.cc:1034] running on GCE node
I20240503 02:11:40.593114 25926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:40.594141 25926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:40.595328 25926 hybrid_clock.cc:648] HybridClock initialized: now 1714702300595291 us; error 28 us; skew 500 ppm
I20240503 02:11:40.595613 25926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:40.601765 25926 webserver.cc:460] Webserver started at http://127.23.104.193:41727/ using document root <none> and password file <none>
I20240503 02:11:40.602443 25926 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:40.602581 25926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:40.602871 25926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:40.605280 25926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3bb103b2fd2344e7b55caf7389acf221"
format_stamp: "Formatted at 2024-05-03 02:11:40 on dist-test-slave-38ht"
I20240503 02:11:40.605923 25926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3bb103b2fd2344e7b55caf7389acf221"
format_stamp: "Formatted at 2024-05-03 02:11:40 on dist-test-slave-38ht"
I20240503 02:11:40.610553 25926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:11:40.613575 25940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:40.614759 25926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:40.614912 25926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3bb103b2fd2344e7b55caf7389acf221"
format_stamp: "Formatted at 2024-05-03 02:11:40 on dist-test-slave-38ht"
I20240503 02:11:40.615089 25926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:40.662366 25926 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:40.663232 25926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:40.663511 25926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:40.664477 25926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:40.665854 25926 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:40.665951 25926 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:40.666060 25926 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:40.666132 25926 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:40.705327 25926 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.193:34143
I20240503 02:11:40.705379 26052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.193:34143 every 8 connection(s)
I20240503 02:11:40.706684 25926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:11:40.713804 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 25926
I20240503 02:11:40.713969 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:11:40.717192 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.194:0
--local_ip_for_outbound_sockets=127.23.104.194
--webserver_interface=127.23.104.194
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--builtin_ntp_servers=127.23.104.212:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:40.719167 26053 heartbeater.cc:347] Connected to a master server at 127.23.104.254:40709
I20240503 02:11:40.719471 26053 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:40.720373 26053 heartbeater.cc:511] Master 127.23.104.254:40709 requested a full tablet report, sending...
I20240503 02:11:40.722460 25867 ts_manager.cc:194] Registered new tserver with Master: 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143)
I20240503 02:11:40.724659 25867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.193:49647
W20240503 02:11:41.034366 26057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:41.034610 26057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:41.034693 26057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:41.041891 26057 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:41.042001 26057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:41.042106 26057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.194
I20240503 02:11:41.050894 26057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.23.104.194
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.194
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:41.052019 26057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:41.053655 26057 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:41.065388 26062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:41.066052 26063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:41.066659 26065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:41.067325 26057 server_base.cc:1034] running on GCE node
I20240503 02:11:41.067874 26057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:41.068679 26057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:41.069864 26057 hybrid_clock.cc:648] HybridClock initialized: now 1714702301069830 us; error 24 us; skew 500 ppm
I20240503 02:11:41.070154 26057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:41.072180 26057 webserver.cc:460] Webserver started at http://127.23.104.194:44019/ using document root <none> and password file <none>
I20240503 02:11:41.072856 26057 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:41.072953 26057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:41.073266 26057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:41.075872 26057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2cf1cac003b44643a2d4de1204aacf0f"
format_stamp: "Formatted at 2024-05-03 02:11:41 on dist-test-slave-38ht"
I20240503 02:11:41.076514 26057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2cf1cac003b44643a2d4de1204aacf0f"
format_stamp: "Formatted at 2024-05-03 02:11:41 on dist-test-slave-38ht"
I20240503 02:11:41.081365 26057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 02:11:41.084390 26071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:41.085557 26057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:11:41.085703 26057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2cf1cac003b44643a2d4de1204aacf0f"
format_stamp: "Formatted at 2024-05-03 02:11:41 on dist-test-slave-38ht"
I20240503 02:11:41.085878 26057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:41.104009 26057 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:41.104735 26057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:41.104982 26057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:41.105813 26057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:41.107151 26057 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:41.107250 26057 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:41.107364 26057 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:41.107436 26057 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:41.137297 26057 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.194:39753
I20240503 02:11:41.137336 26183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.194:39753 every 8 connection(s)
I20240503 02:11:41.138893 26057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:11:41.143973 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 26057
I20240503 02:11:41.144140 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:11:41.146873 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.195:0
--local_ip_for_outbound_sockets=127.23.104.195
--webserver_interface=127.23.104.195
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--builtin_ntp_servers=127.23.104.212:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:41.151170 26184 heartbeater.cc:347] Connected to a master server at 127.23.104.254:40709
I20240503 02:11:41.151487 26184 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:41.152487 26184 heartbeater.cc:511] Master 127.23.104.254:40709 requested a full tablet report, sending...
I20240503 02:11:41.153867 25867 ts_manager.cc:194] Registered new tserver with Master: 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:11:41.154569 25867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.194:48261
W20240503 02:11:41.455039 26187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:41.455276 26187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:41.455397 26187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:41.462505 26187 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:41.462605 26187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:41.462750 26187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.195
I20240503 02:11:41.471132 26187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.23.104.195
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.195
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:41.472308 26187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:41.474011 26187 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:41.485565 26192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:41.486193 26195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:41.486876 26193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:41.487535 26187 server_base.cc:1034] running on GCE node
I20240503 02:11:41.488090 26187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:41.488891 26187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:41.490075 26187 hybrid_clock.cc:648] HybridClock initialized: now 1714702301490001 us; error 68 us; skew 500 ppm
I20240503 02:11:41.490401 26187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:41.496826 26187 webserver.cc:460] Webserver started at http://127.23.104.195:33945/ using document root <none> and password file <none>
I20240503 02:11:41.497761 26187 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:41.497895 26187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:41.498283 26187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:41.501886 26187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7c384005cb3041a0bd70e163473b6acb"
format_stamp: "Formatted at 2024-05-03 02:11:41 on dist-test-slave-38ht"
I20240503 02:11:41.502780 26187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7c384005cb3041a0bd70e163473b6acb"
format_stamp: "Formatted at 2024-05-03 02:11:41 on dist-test-slave-38ht"
I20240503 02:11:41.509224 26187 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20240503 02:11:41.513377 26201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:41.514842 26187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 02:11:41.514998 26187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7c384005cb3041a0bd70e163473b6acb"
format_stamp: "Formatted at 2024-05-03 02:11:41 on dist-test-slave-38ht"
I20240503 02:11:41.515231 26187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:41.563776 26187 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:41.564592 26187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:41.564842 26187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:41.565719 26187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:41.567096 26187 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:41.567191 26187 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:41.567304 26187 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:41.567373 26187 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:41.599334 26187 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.195:45999
I20240503 02:11:41.599392 26313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.195:45999 every 8 connection(s)
I20240503 02:11:41.600605 26187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:11:41.605055 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 26187
I20240503 02:11:41.605244 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:11:41.608889 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.196:0
--local_ip_for_outbound_sockets=127.23.104.196
--webserver_interface=127.23.104.196
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--builtin_ntp_servers=127.23.104.212:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:41.614184 26314 heartbeater.cc:347] Connected to a master server at 127.23.104.254:40709
I20240503 02:11:41.614475 26314 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:41.615406 26314 heartbeater.cc:511] Master 127.23.104.254:40709 requested a full tablet report, sending...
I20240503 02:11:41.616832 25867 ts_manager.cc:194] Registered new tserver with Master: 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999)
I20240503 02:11:41.617558 25867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.195:45833
I20240503 02:11:41.727356 26053 heartbeater.cc:503] Master 127.23.104.254:40709 was elected leader, sending a full tablet report...
W20240503 02:11:41.929921 26317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:41.930191 26317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:41.930342 26317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:41.937095 26317 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:41.937237 26317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:41.937389 26317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.196
I20240503 02:11:41.946095 26317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.23.104.196
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.196
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:41.947273 26317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:41.948918 26317 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:41.960620 26322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:41.961238 26323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:41.961827 26325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:41.962607 26317 server_base.cc:1034] running on GCE node
I20240503 02:11:41.963105 26317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:41.963932 26317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:41.965121 26317 hybrid_clock.cc:648] HybridClock initialized: now 1714702301965071 us; error 46 us; skew 500 ppm
I20240503 02:11:41.965410 26317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:41.967434 26317 webserver.cc:460] Webserver started at http://127.23.104.196:40141/ using document root <none> and password file <none>
I20240503 02:11:41.968160 26317 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:41.968276 26317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:41.968564 26317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:41.970963 26317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "cff7b7a19fc64db5849391b1d8e10cce"
format_stamp: "Formatted at 2024-05-03 02:11:41 on dist-test-slave-38ht"
I20240503 02:11:41.971635 26317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "cff7b7a19fc64db5849391b1d8e10cce"
format_stamp: "Formatted at 2024-05-03 02:11:41 on dist-test-slave-38ht"
I20240503 02:11:41.976130 26317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 02:11:41.979194 26331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:41.980381 26317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:11:41.980525 26317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "cff7b7a19fc64db5849391b1d8e10cce"
format_stamp: "Formatted at 2024-05-03 02:11:41 on dist-test-slave-38ht"
I20240503 02:11:41.980697 26317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:41.992929 26317 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:41.993568 26317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:41.993786 26317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:41.994596 26317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:41.995939 26317 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:41.996034 26317 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:41.996132 26317 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:41.996201 26317 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.028633 26317 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.196:43675
I20240503 02:11:42.028692 26443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.196:43675 every 8 connection(s)
I20240503 02:11:42.029911 26317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:11:42.035578 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 26317
I20240503 02:11:42.035782 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:11:42.038837 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.197:0
--local_ip_for_outbound_sockets=127.23.104.197
--webserver_interface=127.23.104.197
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--builtin_ntp_servers=127.23.104.212:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:42.042390 26444 heartbeater.cc:347] Connected to a master server at 127.23.104.254:40709
I20240503 02:11:42.042656 26444 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:42.043622 26444 heartbeater.cc:511] Master 127.23.104.254:40709 requested a full tablet report, sending...
I20240503 02:11:42.045059 25867 ts_manager.cc:194] Registered new tserver with Master: cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675)
I20240503 02:11:42.045799 25867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.196:54167
I20240503 02:11:42.157120 26184 heartbeater.cc:503] Master 127.23.104.254:40709 was elected leader, sending a full tablet report...
W20240503 02:11:42.348150 26447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:42.348430 26447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:42.348583 26447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:42.355423 26447 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:42.355559 26447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:42.355711 26447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.197
I20240503 02:11:42.364631 26447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.23.104.197
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.197
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:42.365777 26447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:42.367353 26447 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:42.379981 26452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.380585 26453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.381296 26455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:42.385623 26447 server_base.cc:1034] running on GCE node
I20240503 02:11:42.386135 26447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:42.386922 26447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:42.388113 26447 hybrid_clock.cc:648] HybridClock initialized: now 1714702302388064 us; error 40 us; skew 500 ppm
I20240503 02:11:42.388401 26447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:42.390552 26447 webserver.cc:460] Webserver started at http://127.23.104.197:45203/ using document root <none> and password file <none>
I20240503 02:11:42.391242 26447 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:42.391330 26447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:42.391571 26447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:42.393990 26447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b2835e59d40a48aea53c5dedb3bd1e9e"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-38ht"
I20240503 02:11:42.394641 26447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b2835e59d40a48aea53c5dedb3bd1e9e"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-38ht"
I20240503 02:11:42.399298 26447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240503 02:11:42.403084 26461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.406037 26447 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20240503 02:11:42.406275 26447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b2835e59d40a48aea53c5dedb3bd1e9e"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-38ht"
I20240503 02:11:42.406535 26447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:42.448624 26447 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:42.449429 26447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:42.449671 26447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:42.450706 26447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:42.452082 26447 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:42.452186 26447 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.452268 26447 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:42.452329 26447 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.482939 26447 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.197:38347
I20240503 02:11:42.483021 26573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.197:38347 every 8 connection(s)
I20240503 02:11:42.484210 26447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:11:42.487136 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 26447
I20240503 02:11:42.487286 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:11:42.490423 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.198:0
--local_ip_for_outbound_sockets=127.23.104.198
--webserver_interface=127.23.104.198
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--builtin_ntp_servers=127.23.104.212:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:42.497207 26574 heartbeater.cc:347] Connected to a master server at 127.23.104.254:40709
I20240503 02:11:42.497479 26574 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:42.498279 26574 heartbeater.cc:511] Master 127.23.104.254:40709 requested a full tablet report, sending...
I20240503 02:11:42.499539 25867 ts_manager.cc:194] Registered new tserver with Master: b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347)
I20240503 02:11:42.500283 25867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.197:52037
I20240503 02:11:42.620515 26314 heartbeater.cc:503] Master 127.23.104.254:40709 was elected leader, sending a full tablet report...
W20240503 02:11:42.812543 26577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:42.812794 26577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:42.812904 26577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:42.820622 26577 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:42.820746 26577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:42.820914 26577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.198
I20240503 02:11:42.829398 26577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.23.104.198
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.198
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:42.830559 26577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:42.832235 26577 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:42.844201 26582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.844851 26583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.845625 26585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:42.846362 26577 server_base.cc:1034] running on GCE node
I20240503 02:11:42.846933 26577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:42.847757 26577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:42.848994 26577 hybrid_clock.cc:648] HybridClock initialized: now 1714702302848954 us; error 39 us; skew 500 ppm
I20240503 02:11:42.849293 26577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:42.851368 26577 webserver.cc:460] Webserver started at http://127.23.104.198:33435/ using document root <none> and password file <none>
I20240503 02:11:42.852100 26577 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:42.852208 26577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:42.852509 26577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:42.854943 26577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "ae8b30bcfa5b425782a95374787dd230"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-38ht"
I20240503 02:11:42.855741 26577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "ae8b30bcfa5b425782a95374787dd230"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-38ht"
I20240503 02:11:42.860461 26577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:11:42.863327 26591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.864499 26577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:11:42.864647 26577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "ae8b30bcfa5b425782a95374787dd230"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-38ht"
I20240503 02:11:42.864816 26577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:42.887137 26577 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:42.887905 26577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:42.888155 26577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:42.889014 26577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:42.890396 26577 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:42.890482 26577 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.890602 26577 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:42.890682 26577 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.921554 26577 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.198:40369
I20240503 02:11:42.921607 26703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.198:40369 every 8 connection(s)
I20240503 02:11:42.922777 26577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:11:42.927933 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 26577
I20240503 02:11:42.928148 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:11:42.931453 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.199:0
--local_ip_for_outbound_sockets=127.23.104.199
--webserver_interface=127.23.104.199
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--builtin_ntp_servers=127.23.104.212:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:42.935864 26704 heartbeater.cc:347] Connected to a master server at 127.23.104.254:40709
I20240503 02:11:42.936177 26704 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:42.937089 26704 heartbeater.cc:511] Master 127.23.104.254:40709 requested a full tablet report, sending...
I20240503 02:11:42.938374 25867 ts_manager.cc:194] Registered new tserver with Master: ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369)
I20240503 02:11:42.939209 25867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.198:54239
I20240503 02:11:43.048487 26444 heartbeater.cc:503] Master 127.23.104.254:40709 was elected leader, sending a full tablet report...
W20240503 02:11:43.238168 26707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:43.238462 26707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:43.238610 26707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:43.245522 26707 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:43.245647 26707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:43.245801 26707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.199
I20240503 02:11:43.253759 26707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.23.104.199
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.199
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:43.254875 26707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:43.256505 26707 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:43.267732 26713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:43.268498 26712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:43.269104 26715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:43.269860 26707 server_base.cc:1034] running on GCE node
I20240503 02:11:43.270468 26707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:43.271577 26707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:43.272815 26707 hybrid_clock.cc:648] HybridClock initialized: now 1714702303272725 us; error 89 us; skew 500 ppm
I20240503 02:11:43.273092 26707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:43.275164 26707 webserver.cc:460] Webserver started at http://127.23.104.199:33273/ using document root <none> and password file <none>
I20240503 02:11:43.275890 26707 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:43.275993 26707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:43.276284 26707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:43.278738 26707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "0a801f0897884f108cdb89434b18628a"
format_stamp: "Formatted at 2024-05-03 02:11:43 on dist-test-slave-38ht"
I20240503 02:11:43.279412 26707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "0a801f0897884f108cdb89434b18628a"
format_stamp: "Formatted at 2024-05-03 02:11:43 on dist-test-slave-38ht"
I20240503 02:11:43.284058 26707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240503 02:11:43.287184 26721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:43.288460 26707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:43.288590 26707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "0a801f0897884f108cdb89434b18628a"
format_stamp: "Formatted at 2024-05-03 02:11:43 on dist-test-slave-38ht"
I20240503 02:11:43.288777 26707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:43.315670 26707 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:43.316428 26707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:43.316650 26707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:43.317445 26707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:43.318838 26707 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:43.318931 26707 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:43.319026 26707 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:43.319082 26707 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:43.347535 26707 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.199:34279
I20240503 02:11:43.347607 26833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.199:34279 every 8 connection(s)
I20240503 02:11:43.348773 26707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:11:43.359019 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 26707
I20240503 02:11:43.359177 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:11:43.359644 26834 heartbeater.cc:347] Connected to a master server at 127.23.104.254:40709
I20240503 02:11:43.359905 26834 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:43.360807 26834 heartbeater.cc:511] Master 127.23.104.254:40709 requested a full tablet report, sending...
I20240503 02:11:43.362170 25866 ts_manager.cc:194] Registered new tserver with Master: 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279)
I20240503 02:11:43.363058 25866 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.199:33029
I20240503 02:11:43.374159 23971 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 02:11:43.387979 25866 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54512:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 02:11:43.422195 25988 tablet_service.cc:1467] Processing CreateTablet for tablet 5332ecb1e4454eadb47cc721e2fbc487 (DEFAULT_TABLE table=TestTable [id=0dc99a528f224b38bf5d65dde6674a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:43.425483 25988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5332ecb1e4454eadb47cc721e2fbc487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.429175 26769 tablet_service.cc:1467] Processing CreateTablet for tablet 5332ecb1e4454eadb47cc721e2fbc487 (DEFAULT_TABLE table=TestTable [id=0dc99a528f224b38bf5d65dde6674a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:43.430534 26769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5332ecb1e4454eadb47cc721e2fbc487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.430501 26509 tablet_service.cc:1467] Processing CreateTablet for tablet 5332ecb1e4454eadb47cc721e2fbc487 (DEFAULT_TABLE table=TestTable [id=0dc99a528f224b38bf5d65dde6674a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:43.431032 26249 tablet_service.cc:1467] Processing CreateTablet for tablet 5332ecb1e4454eadb47cc721e2fbc487 (DEFAULT_TABLE table=TestTable [id=0dc99a528f224b38bf5d65dde6674a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:43.431823 26509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5332ecb1e4454eadb47cc721e2fbc487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.432248 26249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5332ecb1e4454eadb47cc721e2fbc487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.444612 26857 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb: Bootstrap starting.
I20240503 02:11:43.449582 26857 tablet_bootstrap.cc:654] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.450340 26855 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap starting.
I20240503 02:11:43.451368 26856 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a: Bootstrap starting.
I20240503 02:11:43.452389 26858 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e: Bootstrap starting.
I20240503 02:11:43.455044 26857 log.cc:824] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:43.455879 26858 tablet_bootstrap.cc:654] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.456890 26639 tablet_service.cc:1467] Processing CreateTablet for tablet 5332ecb1e4454eadb47cc721e2fbc487 (DEFAULT_TABLE table=TestTable [id=0dc99a528f224b38bf5d65dde6674a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:43.458142 26858 log.cc:824] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:43.458606 26855 tablet_bootstrap.cc:654] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.458626 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5332ecb1e4454eadb47cc721e2fbc487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.458968 26856 tablet_bootstrap.cc:654] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.461153 26855 log.cc:824] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:43.461959 26856 log.cc:824] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:43.464460 26857 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb: No bootstrap required, opened a new log
I20240503 02:11:43.464735 26857 ts_tablet_manager.cc:1367] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.012s
I20240503 02:11:43.473644 26857 raft_consensus.cc:348] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:43.474242 26857 raft_consensus.cc:374] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.474392 26857 raft_consensus.cc:729] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.474783 26856 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a: No bootstrap required, opened a new log
I20240503 02:11:43.475239 26856 ts_tablet_manager.cc:1367] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.000s
I20240503 02:11:43.475360 26857 consensus_queue.cc:260] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:43.478356 26863 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230: Bootstrap starting.
I20240503 02:11:43.478606 26855 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221: No bootstrap required, opened a new log
I20240503 02:11:43.478984 26855 ts_tablet_manager.cc:1367] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.009s
I20240503 02:11:43.483381 26863 tablet_bootstrap.cc:654] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.483611 26857 ts_tablet_manager.cc:1398] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20240503 02:11:43.484635 26863 log.cc:824] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:43.491444 26858 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e: No bootstrap required, opened a new log
I20240503 02:11:43.492862 26858 ts_tablet_manager.cc:1367] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.026s
I20240503 02:11:43.496932 26863 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230: No bootstrap required, opened a new log
I20240503 02:11:43.497200 26863 ts_tablet_manager.cc:1367] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.005s
I20240503 02:11:43.496982 26856 raft_consensus.cc:348] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:43.497402 26856 raft_consensus.cc:374] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.497514 26856 raft_consensus.cc:729] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a801f0897884f108cdb89434b18628a, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.498173 26856 consensus_queue.cc:260] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:43.501549 26855 raft_consensus.cc:348] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:43.501938 26855 raft_consensus.cc:374] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.502050 26855 raft_consensus.cc:729] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.502321 26574 heartbeater.cc:503] Master 127.23.104.254:40709 was elected leader, sending a full tablet report...
I20240503 02:11:43.502481 26858 raft_consensus.cc:348] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:43.502898 26858 raft_consensus.cc:374] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.503010 26858 raft_consensus.cc:729] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2835e59d40a48aea53c5dedb3bd1e9e, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.503679 26858 consensus_queue.cc:260] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:43.503536 26855 consensus_queue.cc:260] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:43.508008 26863 raft_consensus.cc:348] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:43.508534 26863 raft_consensus.cc:374] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.508644 26863 raft_consensus.cc:729] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.509485 26863 consensus_queue.cc:260] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:43.511391 26834 heartbeater.cc:503] Master 127.23.104.254:40709 was elected leader, sending a full tablet report...
I20240503 02:11:43.511641 26856 ts_tablet_manager.cc:1398] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a: Time spent starting tablet: real 0.036s user 0.024s sys 0.000s
I20240503 02:11:43.512780 26858 ts_tablet_manager.cc:1398] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent starting tablet: real 0.019s user 0.012s sys 0.000s
I20240503 02:11:43.516777 26855 ts_tablet_manager.cc:1398] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221: Time spent starting tablet: real 0.038s user 0.013s sys 0.004s
I20240503 02:11:43.517628 26704 heartbeater.cc:503] Master 127.23.104.254:40709 was elected leader, sending a full tablet report...
I20240503 02:11:43.517918 26863 ts_tablet_manager.cc:1398] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230: Time spent starting tablet: real 0.021s user 0.014s sys 0.007s
I20240503 02:11:43.566929 26867 raft_consensus.cc:482] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:43.567183 26867 raft_consensus.cc:504] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:43.569347 26867 leader_election.cc:288] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369)
I20240503 02:11:43.579933 26529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:11:43.580205 26789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a801f0897884f108cdb89434b18628a" is_pre_election: true
I20240503 02:11:43.580457 26529 raft_consensus.cc:2454] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 0.
I20240503 02:11:43.580682 26789 raft_consensus.cc:2454] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 0.
I20240503 02:11:43.581285 25944 leader_election.cc:302] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0a801f0897884f108cdb89434b18628a, 3bb103b2fd2344e7b55caf7389acf221, b2835e59d40a48aea53c5dedb3bd1e9e; no voters:
I20240503 02:11:43.581737 26867 raft_consensus.cc:2781] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:43.581887 26867 raft_consensus.cc:482] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:43.581981 26867 raft_consensus.cc:3037] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.583372 26269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:11:43.583606 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:11:43.583863 26269 raft_consensus.cc:2454] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 0.
I20240503 02:11:43.584057 26659 raft_consensus.cc:2454] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 0.
I20240503 02:11:43.584877 26867 raft_consensus.cc:504] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:43.585804 26789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a801f0897884f108cdb89434b18628a"
I20240503 02:11:43.585920 26529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e"
I20240503 02:11:43.585976 26269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb"
I20240503 02:11:43.586001 26789 raft_consensus.cc:3037] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.586130 26529 raft_consensus.cc:3037] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.586158 26269 raft_consensus.cc:3037] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.588116 26529 raft_consensus.cc:2454] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 1.
I20240503 02:11:43.588320 26867 leader_election.cc:288] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 1 election: Requested vote from peers 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369)
I20240503 02:11:43.588846 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
I20240503 02:11:43.589018 26659 raft_consensus.cc:3037] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.589095 26789 raft_consensus.cc:2454] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 1.
I20240503 02:11:43.589095 26269 raft_consensus.cc:2454] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 1.
I20240503 02:11:43.589484 25943 leader_election.cc:302] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb, b2835e59d40a48aea53c5dedb3bd1e9e; no voters:
I20240503 02:11:43.589774 26867 raft_consensus.cc:2781] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:43.590137 26867 raft_consensus.cc:686] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 LEADER]: Becoming Leader. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Running, Role: LEADER
I20240503 02:11:43.590682 26867 consensus_queue.cc:237] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:43.591572 26659 raft_consensus.cc:2454] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 1.
I20240503 02:11:43.595037 25865 catalog_manager.cc:5484] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "3bb103b2fd2344e7b55caf7389acf221" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:43.600262 26835 tablet.cc:2369] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:11:43.603367 26315 tablet.cc:2369] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:43.603549 23971 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 02:11:43.605484 23971 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3bb103b2fd2344e7b55caf7389acf221 to finish bootstrapping
I20240503 02:11:43.613190 23971 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2cf1cac003b44643a2d4de1204aacf0f to finish bootstrapping
I20240503 02:11:43.620803 23971 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7c384005cb3041a0bd70e163473b6acb to finish bootstrapping
I20240503 02:11:43.626859 23971 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cff7b7a19fc64db5849391b1d8e10cce to finish bootstrapping
I20240503 02:11:43.634533 23971 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b2835e59d40a48aea53c5dedb3bd1e9e to finish bootstrapping
I20240503 02:11:43.640791 23971 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ae8b30bcfa5b425782a95374787dd230 to finish bootstrapping
I20240503 02:11:43.647240 23971 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0a801f0897884f108cdb89434b18628a to finish bootstrapping
I20240503 02:11:43.653448 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 26707
I20240503 02:11:43.654175 26829 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:11:43.674665 26705 tablet.cc:2369] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:11:43.710687 26054 tablet.cc:2369] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:11:43.736960 26575 tablet.cc:2369] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:43.743060 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 26707
I20240503 02:11:44.031358 26867 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:44.095965 26867 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:44.098114 26882 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:44.115157 25941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.104.199:34279: connect: Connection refused (error 111)
W20240503 02:11:44.123459 25941 consensus_peers.cc:485] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.199:34279: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:46.482376 25941 consensus_peers.cc:485] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.199:34279: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:11:47.510548 25865 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54512:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:11:47.519981 25988 tablet_service.cc:1467] Processing CreateTablet for tablet c3b6c051161d4f9ba9d1d38a06a4c633 (DEFAULT_TABLE table=rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:47.520450 25988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b6c051161d4f9ba9d1d38a06a4c633. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.528548 26509 tablet_service.cc:1467] Processing CreateTablet for tablet c3b6c051161d4f9ba9d1d38a06a4c633 (DEFAULT_TABLE table=rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:47.529096 26509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b6c051161d4f9ba9d1d38a06a4c633. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.532243 26639 tablet_service.cc:1467] Processing CreateTablet for tablet 4a864ac954e046eda8925f5daf4a0f1f (DEFAULT_TABLE table=rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:47.532678 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a864ac954e046eda8925f5daf4a0f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.533779 26508 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d (DEFAULT_TABLE table=rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:47.534130 26508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.536962 26638 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d (DEFAULT_TABLE table=rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:47.537305 26638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.537668 25987 tablet_service.cc:1467] Processing CreateTablet for tablet 4a864ac954e046eda8925f5daf4a0f1f (DEFAULT_TABLE table=rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:47.538179 25987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a864ac954e046eda8925f5daf4a0f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.540623 26249 tablet_service.cc:1467] Processing CreateTablet for tablet c3b6c051161d4f9ba9d1d38a06a4c633 (DEFAULT_TABLE table=rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:47.541128 26249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b6c051161d4f9ba9d1d38a06a4c633. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.544253 26248 tablet_service.cc:1467] Processing CreateTablet for tablet 4a864ac954e046eda8925f5daf4a0f1f (DEFAULT_TABLE table=rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:47.544692 26248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a864ac954e046eda8925f5daf4a0f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.548462 25986 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d (DEFAULT_TABLE table=rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:47.548820 26904 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e: Bootstrap starting.
I20240503 02:11:47.548933 25986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.550515 26904 tablet_bootstrap.cc:654] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:47.552112 26907 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230: Bootstrap starting.
I20240503 02:11:47.553848 26907 tablet_bootstrap.cc:654] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:47.556473 26908 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb: Bootstrap starting.
I20240503 02:11:47.558274 26908 tablet_bootstrap.cc:654] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:47.562546 26118 tablet_service.cc:1467] Processing CreateTablet for tablet 4a864ac954e046eda8925f5daf4a0f1f (DEFAULT_TABLE table=rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:47.562567 26117 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d (DEFAULT_TABLE table=rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:47.564517 26117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.567874 26905 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap starting.
I20240503 02:11:47.562799 26119 tablet_service.cc:1467] Processing CreateTablet for tablet c3b6c051161d4f9ba9d1d38a06a4c633 (DEFAULT_TABLE table=rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:47.568326 26118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a864ac954e046eda8925f5daf4a0f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.569005 26119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b6c051161d4f9ba9d1d38a06a4c633. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.569623 26905 tablet_bootstrap.cc:654] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:47.571421 26907 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230: No bootstrap required, opened a new log
I20240503 02:11:47.571614 26907 ts_tablet_manager.cc:1367] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.008s
I20240503 02:11:47.574023 26907 raft_consensus.cc:348] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } }
I20240503 02:11:47.575501 26907 raft_consensus.cc:374] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:47.575651 26907 raft_consensus.cc:729] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.576016 26907 consensus_queue.cc:260] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } }
I20240503 02:11:47.588831 26378 tablet_service.cc:1467] Processing CreateTablet for tablet 4a864ac954e046eda8925f5daf4a0f1f (DEFAULT_TABLE table=rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:47.590673 26378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a864ac954e046eda8925f5daf4a0f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.591670 26904 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e: No bootstrap required, opened a new log
I20240503 02:11:47.592433 26377 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d (DEFAULT_TABLE table=rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:47.593012 26377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.595922 26904 ts_tablet_manager.cc:1367] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.000s
I20240503 02:11:47.596143 26379 tablet_service.cc:1467] Processing CreateTablet for tablet c3b6c051161d4f9ba9d1d38a06a4c633 (DEFAULT_TABLE table=rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:47.596758 26379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b6c051161d4f9ba9d1d38a06a4c633. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.598188 26904 raft_consensus.cc:348] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } }
I20240503 02:11:47.598656 26904 raft_consensus.cc:374] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:47.598820 26904 raft_consensus.cc:729] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2835e59d40a48aea53c5dedb3bd1e9e, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.599229 26904 consensus_queue.cc:260] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } }
I20240503 02:11:47.602730 26913 tablet_bootstrap.cc:492] T c3b6c051161d4f9ba9d1d38a06a4c633 P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap starting.
I20240503 02:11:47.606251 26913 tablet_bootstrap.cc:654] T c3b6c051161d4f9ba9d1d38a06a4c633 P 2cf1cac003b44643a2d4de1204aacf0f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:47.607398 26908 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb: No bootstrap required, opened a new log
I20240503 02:11:47.607591 26908 ts_tablet_manager.cc:1367] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb: Time spent bootstrapping tablet: real 0.051s user 0.018s sys 0.023s
I20240503 02:11:47.608067 26907 ts_tablet_manager.cc:1398] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230: Time spent starting tablet: real 0.036s user 0.002s sys 0.001s
I20240503 02:11:47.608551 26908 raft_consensus.cc:348] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:47.608775 26908 raft_consensus.cc:374] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:47.608861 26908 raft_consensus.cc:729] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.609104 26908 consensus_queue.cc:260] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:47.609720 26907 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230: Bootstrap starting.
I20240503 02:11:47.609730 26904 ts_tablet_manager.cc:1398] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:11:47.610049 26904 tablet_bootstrap.cc:492] T c3b6c051161d4f9ba9d1d38a06a4c633 P b2835e59d40a48aea53c5dedb3bd1e9e: Bootstrap starting.
I20240503 02:11:47.611419 26907 tablet_bootstrap.cc:654] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:47.611671 26904 tablet_bootstrap.cc:654] T c3b6c051161d4f9ba9d1d38a06a4c633 P b2835e59d40a48aea53c5dedb3bd1e9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:47.617710 26913 log.cc:824] T c3b6c051161d4f9ba9d1d38a06a4c633 P 2cf1cac003b44643a2d4de1204aacf0f: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:47.619386 26908 ts_tablet_manager.cc:1398] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb: Time spent starting tablet: real 0.012s user 0.000s sys 0.004s
I20240503 02:11:47.619976 26908 tablet_bootstrap.cc:492] T c3b6c051161d4f9ba9d1d38a06a4c633 P 7c384005cb3041a0bd70e163473b6acb: Bootstrap starting.
I20240503 02:11:47.621570 26908 tablet_bootstrap.cc:654] T c3b6c051161d4f9ba9d1d38a06a4c633 P 7c384005cb3041a0bd70e163473b6acb: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:47.623261 26904 tablet_bootstrap.cc:492] T c3b6c051161d4f9ba9d1d38a06a4c633 P b2835e59d40a48aea53c5dedb3bd1e9e: No bootstrap required, opened a new log
I20240503 02:11:47.623453 26904 ts_tablet_manager.cc:1367] T c3b6c051161d4f9ba9d1d38a06a4c633 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20240503 02:11:47.624230 26904 raft_consensus.cc:348] T c3b6c051161d4f9ba9d1d38a06a4c633 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:47.624382 26913 tablet_bootstrap.cc:492] T c3b6c051161d4f9ba9d1d38a06a4c633 P 2cf1cac003b44643a2d4de1204aacf0f: No bootstrap required, opened a new log
I20240503 02:11:47.624541 26904 raft_consensus.cc:374] T c3b6c051161d4f9ba9d1d38a06a4c633 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:47.624562 26905 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221: No bootstrap required, opened a new log
I20240503 02:11:47.624644 26913 ts_tablet_manager.cc:1367] T c3b6c051161d4f9ba9d1d38a06a4c633 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20240503 02:11:47.624655 26904 raft_consensus.cc:729] T c3b6c051161d4f9ba9d1d38a06a4c633 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2835e59d40a48aea53c5dedb3bd1e9e, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.624826 26905 ts_tablet_manager.cc:1367] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221: Time spent bootstrapping tablet: real 0.057s user 0.012s sys 0.020s
I20240503 02:11:47.624941 26904 consensus_queue.cc:260] T c3b6c051161d4f9ba9d1d38a06a4c633 P b2835e59d40a48aea53c5dedb3bd1e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:47.625607 26904 ts_tablet_manager.cc:1398] T c3b6c051161d4f9ba9d1d38a06a4c633 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:11:47.625972 26905 raft_consensus.cc:348] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:47.626350 26907 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230: No bootstrap required, opened a new log
I20240503 02:11:47.626360 26905 raft_consensus.cc:374] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:47.626524 26907 ts_tablet_manager.cc:1367] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240503 02:11:47.626564 26905 raft_consensus.cc:729] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.626941 26905 consensus_queue.cc:260] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:47.627758 26905 ts_tablet_manager.cc:1398] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:11:47.627264 26907 raft_consensus.cc:348] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:47.628113 26907 raft_consensus.cc:374] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:47.628194 26907 raft_consensus.cc:729] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.628466 26907 consensus_queue.cc:260] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:47.629016 26907 ts_tablet_manager.cc:1398] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:11:47.629786 26917 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce: Bootstrap starting.
I20240503 02:11:47.630403 26905 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap starting.
I20240503 02:11:47.633913 26917 tablet_bootstrap.cc:654] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:47.633937 26908 tablet_bootstrap.cc:492] T c3b6c051161d4f9ba9d1d38a06a4c633 P 7c384005cb3041a0bd70e163473b6acb: No bootstrap required, opened a new log
I20240503 02:11:47.634078 26908 ts_tablet_manager.cc:1367] T c3b6c051161d4f9ba9d1d38a06a4c633 P 7c384005cb3041a0bd70e163473b6acb: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.005s
I20240503 02:11:47.634370 26905 tablet_bootstrap.cc:654] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:47.634814 26908 raft_consensus.cc:348] T c3b6c051161d4f9ba9d1d38a06a4c633 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:47.635079 26908 raft_consensus.cc:374] T c3b6c051161d4f9ba9d1d38a06a4c633 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:47.635119 26917 log.cc:824] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:47.635169 26908 raft_consensus.cc:729] T c3b6c051161d4f9ba9d1d38a06a4c633 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.636193 26905 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221: No bootstrap required, opened a new log
I20240503 02:11:47.636323 26905 ts_tablet_manager.cc:1367] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 02:11:47.636831 26905 raft_consensus.cc:348] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } }
I20240503 02:11:47.637013 26905 raft_consensus.cc:374] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:47.637091 26905 raft_consensus.cc:729] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.637319 26905 consensus_queue.cc:260] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } }
I20240503 02:11:47.637727 26905 ts_tablet_manager.cc:1398] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:11:47.637956 26905 tablet_bootstrap.cc:492] T c3b6c051161d4f9ba9d1d38a06a4c633 P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap starting.
I20240503 02:11:47.639427 26905 tablet_bootstrap.cc:654] T c3b6c051161d4f9ba9d1d38a06a4c633 P 3bb103b2fd2344e7b55caf7389acf221: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:47.640285 26908 consensus_queue.cc:260] T c3b6c051161d4f9ba9d1d38a06a4c633 P 7c384005cb3041a0bd70e163473b6acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:47.641207 26908 ts_tablet_manager.cc:1398] T c3b6c051161d4f9ba9d1d38a06a4c633 P 7c384005cb3041a0bd70e163473b6acb: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 02:11:47.642812 26905 tablet_bootstrap.cc:492] T c3b6c051161d4f9ba9d1d38a06a4c633 P 3bb103b2fd2344e7b55caf7389acf221: No bootstrap required, opened a new log
I20240503 02:11:47.642993 26905 ts_tablet_manager.cc:1367] T c3b6c051161d4f9ba9d1d38a06a4c633 P 3bb103b2fd2344e7b55caf7389acf221: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:11:47.643607 26917 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce: No bootstrap required, opened a new log
I20240503 02:11:47.643631 26905 raft_consensus.cc:348] T c3b6c051161d4f9ba9d1d38a06a4c633 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:47.643882 26905 raft_consensus.cc:374] T c3b6c051161d4f9ba9d1d38a06a4c633 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:47.644001 26905 raft_consensus.cc:729] T c3b6c051161d4f9ba9d1d38a06a4c633 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.644116 26917 ts_tablet_manager.cc:1367] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20240503 02:11:47.644232 26905 consensus_queue.cc:260] T c3b6c051161d4f9ba9d1d38a06a4c633 P 3bb103b2fd2344e7b55caf7389acf221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:47.644359 26913 raft_consensus.cc:348] T c3b6c051161d4f9ba9d1d38a06a4c633 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:47.644840 26905 ts_tablet_manager.cc:1398] T c3b6c051161d4f9ba9d1d38a06a4c633 P 3bb103b2fd2344e7b55caf7389acf221: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:11:47.644840 26913 raft_consensus.cc:374] T c3b6c051161d4f9ba9d1d38a06a4c633 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:47.645000 26913 raft_consensus.cc:729] T c3b6c051161d4f9ba9d1d38a06a4c633 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.645644 26913 consensus_queue.cc:260] T c3b6c051161d4f9ba9d1d38a06a4c633 P 2cf1cac003b44643a2d4de1204aacf0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:47.647203 26913 ts_tablet_manager.cc:1398] T c3b6c051161d4f9ba9d1d38a06a4c633 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent starting tablet: real 0.022s user 0.011s sys 0.000s
I20240503 02:11:47.647634 26913 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap starting.
I20240503 02:11:47.649211 26913 tablet_bootstrap.cc:654] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:47.651213 26913 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f: No bootstrap required, opened a new log
I20240503 02:11:47.651348 26913 ts_tablet_manager.cc:1367] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:11:47.652195 26913 raft_consensus.cc:348] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:47.652423 26913 raft_consensus.cc:374] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:47.652513 26913 raft_consensus.cc:729] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.652786 26913 consensus_queue.cc:260] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:47.653321 26913 ts_tablet_manager.cc:1398] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:47.653260 26917 raft_consensus.cc:348] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:47.653599 26913 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap starting.
I20240503 02:11:47.653743 26917 raft_consensus.cc:374] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:47.653849 26917 raft_consensus.cc:729] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.654352 26912 raft_consensus.cc:482] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:47.654533 26912 raft_consensus.cc:504] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } }
I20240503 02:11:47.654675 26917 consensus_queue.cc:260] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:47.655069 26913 tablet_bootstrap.cc:654] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:47.656668 26912 leader_election.cc:288] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143)
I20240503 02:11:47.656836 26913 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f: No bootstrap required, opened a new log
I20240503 02:11:47.656976 26913 ts_tablet_manager.cc:1367] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240503 02:11:47.657426 26917 ts_tablet_manager.cc:1398] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce: Time spent starting tablet: real 0.013s user 0.010s sys 0.001s
I20240503 02:11:47.657799 26917 tablet_bootstrap.cc:492] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce: Bootstrap starting.
I20240503 02:11:47.657792 26913 raft_consensus.cc:348] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } }
I20240503 02:11:47.657992 26913 raft_consensus.cc:374] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:47.658056 26913 raft_consensus.cc:729] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.658365 26913 consensus_queue.cc:260] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } }
I20240503 02:11:47.658991 26913 ts_tablet_manager.cc:1398] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:11:47.659292 26917 tablet_bootstrap.cc:654] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:47.659662 26914 raft_consensus.cc:482] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:47.659865 26914 raft_consensus.cc:504] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } }
I20240503 02:11:47.662307 26914 leader_election.cc:288] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143)
I20240503 02:11:47.687980 26399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
W20240503 02:11:47.689574 26464 leader_election.cc:341] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:47.696830 26917 tablet_bootstrap.cc:492] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce: No bootstrap required, opened a new log
I20240503 02:11:47.697098 26917 ts_tablet_manager.cc:1367] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.037s
I20240503 02:11:47.698417 26917 raft_consensus.cc:348] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:47.698791 26917 raft_consensus.cc:374] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:47.698944 26917 raft_consensus.cc:729] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.699263 26917 consensus_queue.cc:260] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:47.700090 26917 ts_tablet_manager.cc:1398] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 02:11:47.700467 26917 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce: Bootstrap starting.
I20240503 02:11:47.702690 26917 tablet_bootstrap.cc:654] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:47.717743 26917 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce: No bootstrap required, opened a new log
I20240503 02:11:47.717890 26917 ts_tablet_manager.cc:1367] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.002s
I20240503 02:11:47.718546 26917 raft_consensus.cc:348] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } }
I20240503 02:11:47.718781 26917 raft_consensus.cc:374] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:47.718854 26917 raft_consensus.cc:729] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.719097 26917 consensus_queue.cc:260] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } }
I20240503 02:11:47.719645 26917 ts_tablet_manager.cc:1398] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:11:47.726035 26529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:11:47.726521 26529 raft_consensus.cc:2454] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 0.
I20240503 02:11:47.734743 26139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:11:47.734979 26399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:11:47.734946 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:11:47.735256 26139 raft_consensus.cc:2454] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 0.
I20240503 02:11:47.735276 26399 raft_consensus.cc:2454] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 0.
I20240503 02:11:47.735375 26008 raft_consensus.cc:2454] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 0.
I20240503 02:11:47.735723 26594 leader_election.cc:302] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: ae8b30bcfa5b425782a95374787dd230, b2835e59d40a48aea53c5dedb3bd1e9e, cff7b7a19fc64db5849391b1d8e10cce; no voters:
I20240503 02:11:47.736250 26912 raft_consensus.cc:2781] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:47.736357 26912 raft_consensus.cc:482] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:47.736423 26912 raft_consensus.cc:3037] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:47.738799 26462 leader_election.cc:302] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, 3bb103b2fd2344e7b55caf7389acf221, b2835e59d40a48aea53c5dedb3bd1e9e; no voters: cff7b7a19fc64db5849391b1d8e10cce
I20240503 02:11:47.739226 26914 raft_consensus.cc:2781] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:47.739346 26914 raft_consensus.cc:482] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:47.739430 26914 raft_consensus.cc:3037] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:47.739727 26912 raft_consensus.cc:504] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } }
I20240503 02:11:47.740720 26912 leader_election.cc:288] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 1 election: Requested vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143)
I20240503 02:11:47.742053 26139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:11:47.742226 26139 raft_consensus.cc:2454] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 0.
I20240503 02:11:47.742220 26914 raft_consensus.cc:504] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } }
I20240503 02:11:47.742475 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:11:47.742698 26008 raft_consensus.cc:2454] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 0.
I20240503 02:11:47.742586 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221"
I20240503 02:11:47.742877 26007 raft_consensus.cc:3037] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:47.743718 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f"
I20240503 02:11:47.743970 26138 raft_consensus.cc:3037] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:47.745637 26007 raft_consensus.cc:2454] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 1.
I20240503 02:11:47.746698 26138 raft_consensus.cc:2454] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 1.
I20240503 02:11:47.747076 26592 leader_election.cc:302] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, 3bb103b2fd2344e7b55caf7389acf221, ae8b30bcfa5b425782a95374787dd230; no voters:
I20240503 02:11:47.747371 26399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce"
I20240503 02:11:47.747570 26399 raft_consensus.cc:3037] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:47.747769 26941 raft_consensus.cc:2781] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:47.749152 26398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce"
I20240503 02:11:47.749625 26914 leader_election.cc:288] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 election: Requested vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143)
I20240503 02:11:47.750430 26399 raft_consensus.cc:2454] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 1.
I20240503 02:11:47.750806 26529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e"
I20240503 02:11:47.751063 26529 raft_consensus.cc:2379] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ae8b30bcfa5b425782a95374787dd230 in current term 1: Already voted for candidate b2835e59d40a48aea53c5dedb3bd1e9e in this term.
I20240503 02:11:47.751585 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221"
I20240503 02:11:47.751888 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f"
I20240503 02:11:47.752148 26138 raft_consensus.cc:2379] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b2835e59d40a48aea53c5dedb3bd1e9e in current term 1: Already voted for candidate ae8b30bcfa5b425782a95374787dd230 in this term.
I20240503 02:11:47.752389 26007 raft_consensus.cc:2379] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b2835e59d40a48aea53c5dedb3bd1e9e in current term 1: Already voted for candidate ae8b30bcfa5b425782a95374787dd230 in this term.
I20240503 02:11:47.752844 26912 raft_consensus.cc:482] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:47.753000 26912 raft_consensus.cc:504] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:47.753989 26912 leader_election.cc:288] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:11:47.754462 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:11:47.754645 26007 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 0.
I20240503 02:11:47.755553 26941 raft_consensus.cc:686] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 1 LEADER]: Becoming Leader. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Running, Role: LEADER
I20240503 02:11:47.756390 26399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:11:47.756214 26941 consensus_queue.cc:237] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } }
I20240503 02:11:47.756592 26399 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 0.
I20240503 02:11:47.756893 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:11:47.757097 26138 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 0.
I20240503 02:11:47.757431 26592 leader_election.cc:302] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, 3bb103b2fd2344e7b55caf7389acf221, ae8b30bcfa5b425782a95374787dd230; no voters:
I20240503 02:11:47.757753 26942 raft_consensus.cc:2781] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:47.757869 26942 raft_consensus.cc:482] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:47.757999 26942 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:47.761889 26942 raft_consensus.cc:504] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:47.762779 26942 leader_election.cc:288] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 1 election: Requested vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:11:47.763159 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221"
I20240503 02:11:47.763330 26007 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:47.765857 26007 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 1.
I20240503 02:11:47.766371 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f"
I20240503 02:11:47.766563 26138 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:47.768143 26269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:11:47.768384 26269 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 0.
I20240503 02:11:47.768519 26399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce"
I20240503 02:11:47.768687 26399 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:47.768775 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb"
I20240503 02:11:47.769008 26268 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:47.771330 26399 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 1.
I20240503 02:11:47.771874 26594 leader_election.cc:302] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, ae8b30bcfa5b425782a95374787dd230, cff7b7a19fc64db5849391b1d8e10cce; no voters:
I20240503 02:11:47.772310 26268 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 1.
I20240503 02:11:47.772686 26942 raft_consensus.cc:2781] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:47.772939 26942 raft_consensus.cc:686] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 1 LEADER]: Becoming Leader. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Running, Role: LEADER
I20240503 02:11:47.773259 26942 consensus_queue.cc:237] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:47.774646 26138 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 1.
I20240503 02:11:47.776415 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:11:47.776988 26658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
I20240503 02:11:47.778751 26465 leader_election.cc:302] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: b2835e59d40a48aea53c5dedb3bd1e9e, cff7b7a19fc64db5849391b1d8e10cce; no voters: 2cf1cac003b44643a2d4de1204aacf0f, 3bb103b2fd2344e7b55caf7389acf221, ae8b30bcfa5b425782a95374787dd230
I20240503 02:11:47.779034 26914 raft_consensus.cc:2726] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:11:47.779784 25865 catalog_manager.cc:5484] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae8b30bcfa5b425782a95374787dd230 (127.23.104.198). New cstate: current_term: 1 leader_uuid: "ae8b30bcfa5b425782a95374787dd230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:47.784024 25865 catalog_manager.cc:5484] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae8b30bcfa5b425782a95374787dd230 (127.23.104.198). New cstate: current_term: 1 leader_uuid: "ae8b30bcfa5b425782a95374787dd230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:47.785600 26445 tablet.cc:2369] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:47.846786 26926 raft_consensus.cc:482] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:47.846956 26926 raft_consensus.cc:504] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:47.849030 26926 leader_election.cc:288] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347)
I20240503 02:11:47.865425 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b6c051161d4f9ba9d1d38a06a4c633" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:11:47.865676 26007 raft_consensus.cc:2454] T c3b6c051161d4f9ba9d1d38a06a4c633 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 0.
I20240503 02:11:47.865917 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b6c051161d4f9ba9d1d38a06a4c633" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:11:47.866132 26268 raft_consensus.cc:2454] T c3b6c051161d4f9ba9d1d38a06a4c633 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 0.
I20240503 02:11:47.866490 26334 leader_election.cc:302] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb, cff7b7a19fc64db5849391b1d8e10cce; no voters:
I20240503 02:11:47.866853 26926 raft_consensus.cc:2781] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:47.866986 26926 raft_consensus.cc:482] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:47.867090 26926 raft_consensus.cc:3037] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:47.868757 26529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b6c051161d4f9ba9d1d38a06a4c633" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:11:47.868999 26529 raft_consensus.cc:2454] T c3b6c051161d4f9ba9d1d38a06a4c633 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 0.
I20240503 02:11:47.869170 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b6c051161d4f9ba9d1d38a06a4c633" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:11:47.869405 26138 raft_consensus.cc:2454] T c3b6c051161d4f9ba9d1d38a06a4c633 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 0.
I20240503 02:11:47.870039 26926 raft_consensus.cc:504] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:47.871039 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b6c051161d4f9ba9d1d38a06a4c633" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221"
I20240503 02:11:47.871209 26926 leader_election.cc:288] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 election: Requested vote from peers 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347)
I20240503 02:11:47.871214 26007 raft_consensus.cc:3037] T c3b6c051161d4f9ba9d1d38a06a4c633 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:47.871301 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b6c051161d4f9ba9d1d38a06a4c633" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb"
I20240503 02:11:47.871479 26268 raft_consensus.cc:3037] T c3b6c051161d4f9ba9d1d38a06a4c633 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:47.871469 26529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b6c051161d4f9ba9d1d38a06a4c633" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e"
I20240503 02:11:47.871498 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b6c051161d4f9ba9d1d38a06a4c633" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f"
I20240503 02:11:47.871649 26529 raft_consensus.cc:3037] T c3b6c051161d4f9ba9d1d38a06a4c633 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:47.871654 26138 raft_consensus.cc:3037] T c3b6c051161d4f9ba9d1d38a06a4c633 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:47.874133 26268 raft_consensus.cc:2454] T c3b6c051161d4f9ba9d1d38a06a4c633 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 1.
I20240503 02:11:47.874217 26529 raft_consensus.cc:2454] T c3b6c051161d4f9ba9d1d38a06a4c633 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 1.
I20240503 02:11:47.874274 26007 raft_consensus.cc:2454] T c3b6c051161d4f9ba9d1d38a06a4c633 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 1.
I20240503 02:11:47.874274 26138 raft_consensus.cc:2454] T c3b6c051161d4f9ba9d1d38a06a4c633 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 1.
I20240503 02:11:47.874756 26332 leader_election.cc:302] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, b2835e59d40a48aea53c5dedb3bd1e9e, cff7b7a19fc64db5849391b1d8e10cce; no voters:
I20240503 02:11:47.875087 26926 raft_consensus.cc:2781] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:47.875475 26926 raft_consensus.cc:686] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [term 1 LEADER]: Becoming Leader. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Running, Role: LEADER
I20240503 02:11:47.875969 26926 consensus_queue.cc:237] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:47.882055 25865 catalog_manager.cc:5484] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: term changed from 0 to 1, leader changed from <none> to cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196). New cstate: current_term: 1 leader_uuid: "cff7b7a19fc64db5849391b1d8e10cce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:47.894629 26185 tablet.cc:2369] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:48.138352 26941 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:48.180965 26941 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:48.230595 26941 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:48.237759 26942 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:48.238271 26942 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:48.262792 26912 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:48.269196 26912 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:48.270613 26942 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:48.276697 26952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:48.277092 26952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:48.286128 26952 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:11:48.295151 26926 consensus_queue.cc:1035] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 02:11:48.323966 26926 consensus_queue.cc:1035] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:48.331099 26926 consensus_queue.cc:1035] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:48.342240 26986 consensus_queue.cc:1035] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 322.72 ms
time per row: 0.16136 ms
I20240503 02:11:48.786648 25864 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54512:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:11:48.795697 26248 tablet_service.cc:1467] Processing CreateTablet for tablet 508ee4b913dd40b49d50be3fa4cdd6c4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:48.795986 26638 tablet_service.cc:1467] Processing CreateTablet for tablet 508ee4b913dd40b49d50be3fa4cdd6c4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:48.796205 26248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508ee4b913dd40b49d50be3fa4cdd6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.796422 26638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508ee4b913dd40b49d50be3fa4cdd6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.798163 26119 tablet_service.cc:1467] Processing CreateTablet for tablet 508ee4b913dd40b49d50be3fa4cdd6c4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:48.798588 26119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508ee4b913dd40b49d50be3fa4cdd6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.800190 26378 tablet_service.cc:1467] Processing CreateTablet for tablet 508ee4b913dd40b49d50be3fa4cdd6c4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:48.800518 26378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508ee4b913dd40b49d50be3fa4cdd6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.802356 27034 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb: Bootstrap starting.
I20240503 02:11:48.802624 26117 tablet_service.cc:1467] Processing CreateTablet for tablet 041ffbedf2c04dbea4661b4c33f78ac1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:48.802937 26117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041ffbedf2c04dbea4661b4c33f78ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.803994 27034 tablet_bootstrap.cc:654] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:48.804143 26377 tablet_service.cc:1467] Processing CreateTablet for tablet 041ffbedf2c04dbea4661b4c33f78ac1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:48.804504 26377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041ffbedf2c04dbea4661b4c33f78ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.805851 27034 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb: No bootstrap required, opened a new log
I20240503 02:11:48.805997 27034 ts_tablet_manager.cc:1367] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:11:48.806691 27034 raft_consensus.cc:348] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.806907 27034 raft_consensus.cc:374] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:48.807009 27034 raft_consensus.cc:729] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Initialized, Role: FOLLOWER
I20240503 02:11:48.807086 26118 tablet_service.cc:1467] Processing CreateTablet for tablet 495c99a8850d4ec6b52a89bec172bab0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:48.807253 27034 consensus_queue.cc:260] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.807432 26118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495c99a8850d4ec6b52a89bec172bab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.807534 27036 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap starting.
I20240503 02:11:48.807775 27034 ts_tablet_manager.cc:1398] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:11:48.807991 25987 tablet_service.cc:1467] Processing CreateTablet for tablet 495c99a8850d4ec6b52a89bec172bab0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:48.808368 25987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495c99a8850d4ec6b52a89bec172bab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.809528 27036 tablet_bootstrap.cc:654] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:48.811582 27036 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f: No bootstrap required, opened a new log
I20240503 02:11:48.811726 27036 ts_tablet_manager.cc:1367] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:11:48.812425 27036 raft_consensus.cc:348] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.812647 27036 raft_consensus.cc:374] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:48.812736 27036 raft_consensus.cc:729] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Initialized, Role: FOLLOWER
I20240503 02:11:48.813093 26639 tablet_service.cc:1467] Processing CreateTablet for tablet 041ffbedf2c04dbea4661b4c33f78ac1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:48.813444 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041ffbedf2c04dbea4661b4c33f78ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.813699 26508 tablet_service.cc:1467] Processing CreateTablet for tablet 508ee4b913dd40b49d50be3fa4cdd6c4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:48.814077 26508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508ee4b913dd40b49d50be3fa4cdd6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.813994 27036 consensus_queue.cc:260] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.814985 27036 ts_tablet_manager.cc:1398] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:48.816139 26509 tablet_service.cc:1467] Processing CreateTablet for tablet 041ffbedf2c04dbea4661b4c33f78ac1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:48.816190 26507 tablet_service.cc:1467] Processing CreateTablet for tablet 495c99a8850d4ec6b52a89bec172bab0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:48.816504 26509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041ffbedf2c04dbea4661b4c33f78ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.817135 26507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495c99a8850d4ec6b52a89bec172bab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.820631 26248 tablet_service.cc:1467] Processing CreateTablet for tablet 041ffbedf2c04dbea4661b4c33f78ac1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:48.820986 26248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041ffbedf2c04dbea4661b4c33f78ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.823192 26924 raft_consensus.cc:482] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:48.823330 26924 raft_consensus.cc:504] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.824699 26249 tablet_service.cc:1467] Processing CreateTablet for tablet 495c99a8850d4ec6b52a89bec172bab0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:48.825060 26249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495c99a8850d4ec6b52a89bec172bab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.827706 27039 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce: Bootstrap starting.
I20240503 02:11:48.828009 27036 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap starting.
I20240503 02:11:48.829485 27039 tablet_bootstrap.cc:654] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:48.829556 27036 tablet_bootstrap.cc:654] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:48.831194 27038 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230: Bootstrap starting.
I20240503 02:11:48.831624 26924 leader_election.cc:288] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347)
I20240503 02:11:48.833317 27038 tablet_bootstrap.cc:654] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:48.835054 27041 tablet_bootstrap.cc:492] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e: Bootstrap starting.
I20240503 02:11:48.836301 27034 tablet_bootstrap.cc:492] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb: Bootstrap starting.
I20240503 02:11:48.836819 27041 tablet_bootstrap.cc:654] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:48.837891 27034 tablet_bootstrap.cc:654] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:48.838482 27039 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce: No bootstrap required, opened a new log
I20240503 02:11:48.838619 27039 ts_tablet_manager.cc:1367] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240503 02:11:48.839248 27036 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f: No bootstrap required, opened a new log
I20240503 02:11:48.839416 27036 ts_tablet_manager.cc:1367] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 02:11:48.839373 27039 raft_consensus.cc:348] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.839624 27039 raft_consensus.cc:374] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:48.839712 27039 raft_consensus.cc:729] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Initialized, Role: FOLLOWER
I20240503 02:11:48.839972 27039 consensus_queue.cc:260] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.840188 27036 raft_consensus.cc:348] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.840407 27036 raft_consensus.cc:374] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:48.840351 26378 tablet_service.cc:1467] Processing CreateTablet for tablet 495c99a8850d4ec6b52a89bec172bab0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:48.840471 27036 raft_consensus.cc:729] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Initialized, Role: FOLLOWER
I20240503 02:11:48.840694 27036 consensus_queue.cc:260] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.840873 27039 ts_tablet_manager.cc:1398] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:48.841128 26378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495c99a8850d4ec6b52a89bec172bab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.841703 27036 ts_tablet_manager.cc:1398] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:48.841962 27036 tablet_bootstrap.cc:492] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap starting.
I20240503 02:11:48.843324 27036 tablet_bootstrap.cc:654] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:48.845268 27041 tablet_bootstrap.cc:492] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e: No bootstrap required, opened a new log
I20240503 02:11:48.845410 27041 ts_tablet_manager.cc:1367] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240503 02:11:48.845916 27039 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce: Bootstrap starting.
I20240503 02:11:48.846315 27041 raft_consensus.cc:348] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:48.846583 27041 raft_consensus.cc:374] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:48.846688 27041 raft_consensus.cc:729] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2835e59d40a48aea53c5dedb3bd1e9e, State: Initialized, Role: FOLLOWER
I20240503 02:11:48.847375 27041 consensus_queue.cc:260] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:48.848111 27039 tablet_bootstrap.cc:654] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:48.849200 27034 tablet_bootstrap.cc:492] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb: No bootstrap required, opened a new log
I20240503 02:11:48.849404 27034 ts_tablet_manager.cc:1367] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20240503 02:11:48.850023 27038 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230: No bootstrap required, opened a new log
I20240503 02:11:48.850209 27038 ts_tablet_manager.cc:1367] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.002s
I20240503 02:11:48.850606 27034 raft_consensus.cc:348] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:48.850935 27034 raft_consensus.cc:374] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:48.851071 27034 raft_consensus.cc:729] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Initialized, Role: FOLLOWER
I20240503 02:11:48.851154 27038 raft_consensus.cc:348] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.851462 27038 raft_consensus.cc:374] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:48.851567 27038 raft_consensus.cc:729] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Initialized, Role: FOLLOWER
I20240503 02:11:48.851423 27034 consensus_queue.cc:260] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:48.851891 27038 consensus_queue.cc:260] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.852516 27038 ts_tablet_manager.cc:1398] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:48.853020 27034 ts_tablet_manager.cc:1398] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:48.853494 27038 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230: Bootstrap starting.
I20240503 02:11:48.855772 27038 tablet_bootstrap.cc:654] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:48.857573 27039 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce: No bootstrap required, opened a new log
I20240503 02:11:48.857756 27039 ts_tablet_manager.cc:1367] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 02:11:48.858229 27034 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb: Bootstrap starting.
I20240503 02:11:48.858844 27039 raft_consensus.cc:348] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.859138 27039 raft_consensus.cc:374] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:48.859230 27039 raft_consensus.cc:729] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Initialized, Role: FOLLOWER
I20240503 02:11:48.859545 27039 consensus_queue.cc:260] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.860282 27034 tablet_bootstrap.cc:654] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:48.863278 27039 ts_tablet_manager.cc:1398] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:11:48.867051 27050 raft_consensus.cc:482] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:48.867213 27036 tablet_bootstrap.cc:492] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f: No bootstrap required, opened a new log
I20240503 02:11:48.867369 27036 ts_tablet_manager.cc:1367] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.003s
I20240503 02:11:48.867218 27050 raft_consensus.cc:504] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:48.868815 27034 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb: No bootstrap required, opened a new log
I20240503 02:11:48.869062 27034 ts_tablet_manager.cc:1367] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240503 02:11:48.869017 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495c99a8850d4ec6b52a89bec172bab0" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
W20240503 02:11:48.869935 26462 leader_election.cc:341] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:48.870394 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495c99a8850d4ec6b52a89bec172bab0" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:11:48.869963 27034 raft_consensus.cc:348] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
W20240503 02:11:48.871263 26462 leader_election.cc:341] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:48.871301 27034 raft_consensus.cc:374] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:48.871469 27034 raft_consensus.cc:729] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Initialized, Role: FOLLOWER
I20240503 02:11:48.871850 27034 consensus_queue.cc:260] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.872738 27034 ts_tablet_manager.cc:1398] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:48.868541 27050 leader_election.cc:288] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:11:48.876008 27038 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230: No bootstrap required, opened a new log
I20240503 02:11:48.876190 27038 ts_tablet_manager.cc:1367] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240503 02:11:48.878744 27038 raft_consensus.cc:348] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.879091 27038 raft_consensus.cc:374] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:48.879186 27038 raft_consensus.cc:729] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Initialized, Role: FOLLOWER
I20240503 02:11:48.879474 27038 consensus_queue.cc:260] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.880306 27041 ts_tablet_manager.cc:1398] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent starting tablet: real 0.035s user 0.000s sys 0.007s
I20240503 02:11:48.880628 27041 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e: Bootstrap starting.
I20240503 02:11:48.880686 27038 ts_tablet_manager.cc:1398] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:11:48.882184 27041 tablet_bootstrap.cc:654] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:48.884904 27036 raft_consensus.cc:348] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:48.885206 27036 raft_consensus.cc:374] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:48.885313 27036 raft_consensus.cc:729] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Initialized, Role: FOLLOWER
I20240503 02:11:48.885540 27036 consensus_queue.cc:260] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:48.889175 27036 ts_tablet_manager.cc:1398] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20240503 02:11:48.890334 26398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041ffbedf2c04dbea4661b4c33f78ac1" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:11:48.890589 26398 raft_consensus.cc:2454] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 0.
W20240503 02:11:48.891548 26464 leader_election.cc:341] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675): Not found: Tablet not found: 495c99a8850d4ec6b52a89bec172bab0
I20240503 02:11:48.891794 26464 leader_election.cc:302] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: b2835e59d40a48aea53c5dedb3bd1e9e; no voters: 2cf1cac003b44643a2d4de1204aacf0f, 3bb103b2fd2344e7b55caf7389acf221, cff7b7a19fc64db5849391b1d8e10cce
I20240503 02:11:48.892161 27050 raft_consensus.cc:2726] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:11:48.892894 27039 tablet_bootstrap.cc:492] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce: Bootstrap starting.
I20240503 02:11:48.893566 26658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041ffbedf2c04dbea4661b4c33f78ac1" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:11:48.893813 26658 raft_consensus.cc:2454] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 0.
I20240503 02:11:48.893978 27040 tablet_bootstrap.cc:492] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap starting.
I20240503 02:11:48.894948 27039 tablet_bootstrap.cc:654] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:48.895673 27040 tablet_bootstrap.cc:654] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:48.895992 26075 leader_election.cc:302] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, ae8b30bcfa5b425782a95374787dd230, cff7b7a19fc64db5849391b1d8e10cce; no voters:
I20240503 02:11:48.896812 26924 raft_consensus.cc:2781] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:48.896930 26924 raft_consensus.cc:482] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:48.897024 26924 raft_consensus.cc:3037] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:48.898057 27041 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e: No bootstrap required, opened a new log
I20240503 02:11:48.898281 27041 ts_tablet_manager.cc:1367] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240503 02:11:48.899057 26924 raft_consensus.cc:504] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.899276 27041 raft_consensus.cc:348] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.899618 27041 raft_consensus.cc:374] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:48.899717 27041 raft_consensus.cc:729] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2835e59d40a48aea53c5dedb3bd1e9e, State: Initialized, Role: FOLLOWER
I20240503 02:11:48.899919 26924 leader_election.cc:288] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 election: Requested vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347)
I20240503 02:11:48.900194 26398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041ffbedf2c04dbea4661b4c33f78ac1" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce"
I20240503 02:11:48.900414 26398 raft_consensus.cc:3037] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:48.900512 26658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041ffbedf2c04dbea4661b4c33f78ac1" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
I20240503 02:11:48.900708 26658 raft_consensus.cc:3037] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:48.903352 26398 raft_consensus.cc:2454] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:48.903450 26658 raft_consensus.cc:2454] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:48.904482 26075 leader_election.cc:302] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, ae8b30bcfa5b425782a95374787dd230, cff7b7a19fc64db5849391b1d8e10cce; no voters:
I20240503 02:11:48.904750 26924 raft_consensus.cc:2781] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:48.908072 27041 consensus_queue.cc:260] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.908844 27041 ts_tablet_manager.cc:1398] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20240503 02:11:48.908958 26955 raft_consensus.cc:482] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:48.909209 27041 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e: Bootstrap starting.
I20240503 02:11:48.909147 26955 raft_consensus.cc:504] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:48.909682 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495c99a8850d4ec6b52a89bec172bab0" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:11:48.911600 27041 tablet_bootstrap.cc:654] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:48.916378 26268 raft_consensus.cc:2454] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 0.
I20240503 02:11:48.921121 27041 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e: No bootstrap required, opened a new log
I20240503 02:11:48.921286 27041 ts_tablet_manager.cc:1367] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240503 02:11:48.922120 27041 raft_consensus.cc:348] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.922349 27041 raft_consensus.cc:374] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:48.922436 27041 raft_consensus.cc:729] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2835e59d40a48aea53c5dedb3bd1e9e, State: Initialized, Role: FOLLOWER
I20240503 02:11:48.922665 27041 consensus_queue.cc:260] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.924006 27041 ts_tablet_manager.cc:1398] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:11:48.924058 26955 leader_election.cc:288] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:11:48.943331 26528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041ffbedf2c04dbea4661b4c33f78ac1" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:11:48.943589 26528 raft_consensus.cc:2454] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 0.
I20240503 02:11:48.944387 26528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041ffbedf2c04dbea4661b4c33f78ac1" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e"
I20240503 02:11:48.944610 26528 raft_consensus.cc:3037] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:48.948318 26528 raft_consensus.cc:2454] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:48.973573 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041ffbedf2c04dbea4661b4c33f78ac1" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:11:48.973893 26268 raft_consensus.cc:2454] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 0.
I20240503 02:11:48.975256 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041ffbedf2c04dbea4661b4c33f78ac1" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb"
I20240503 02:11:48.975477 26268 raft_consensus.cc:3037] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:48.979902 26268 raft_consensus.cc:2454] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:48.984977 27053 raft_consensus.cc:482] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:48.985147 27053 raft_consensus.cc:504] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.986253 27053 leader_election.cc:288] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675)
I20240503 02:11:48.986546 27069 raft_consensus.cc:482] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:48.986773 27069 raft_consensus.cc:504] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:48.987244 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508ee4b913dd40b49d50be3fa4cdd6c4" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:11:48.987525 26138 raft_consensus.cc:2454] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 0.
I20240503 02:11:48.988351 26658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508ee4b913dd40b49d50be3fa4cdd6c4" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:11:48.988685 26658 raft_consensus.cc:2454] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 0.
I20240503 02:11:48.988822 27069 leader_election.cc:288] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675)
I20240503 02:11:48.989434 26398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495c99a8850d4ec6b52a89bec172bab0" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:11:48.991822 26465 leader_election.cc:302] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, ae8b30bcfa5b425782a95374787dd230, b2835e59d40a48aea53c5dedb3bd1e9e; no voters:
I20240503 02:11:48.992206 27053 raft_consensus.cc:2781] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:48.992339 27053 raft_consensus.cc:482] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:48.992442 27053 raft_consensus.cc:3037] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:48.995231 26528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495c99a8850d4ec6b52a89bec172bab0" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:11:48.995522 26528 raft_consensus.cc:2454] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 0.
I20240503 02:11:48.996737 26398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508ee4b913dd40b49d50be3fa4cdd6c4" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:11:48.997053 26398 raft_consensus.cc:2454] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 0.
I20240503 02:11:48.997951 27040 tablet_bootstrap.cc:492] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221: No bootstrap required, opened a new log
I20240503 02:11:48.998165 27040 ts_tablet_manager.cc:1367] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221: Time spent bootstrapping tablet: real 0.104s user 0.003s sys 0.001s
I20240503 02:11:48.998621 27053 raft_consensus.cc:504] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:48.999230 26269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495c99a8850d4ec6b52a89bec172bab0" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:11:48.999226 27040 raft_consensus.cc:348] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:48.999500 26269 raft_consensus.cc:2454] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 0.
I20240503 02:11:48.999572 27040 raft_consensus.cc:374] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:48.999668 27040 raft_consensus.cc:729] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Initialized, Role: FOLLOWER
I20240503 02:11:48.999990 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508ee4b913dd40b49d50be3fa4cdd6c4" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:11:49.000607 26268 raft_consensus.cc:2454] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 0.
I20240503 02:11:49.001691 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508ee4b913dd40b49d50be3fa4cdd6c4" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f"
I20240503 02:11:49.001919 26138 raft_consensus.cc:3037] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:49.002686 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508ee4b913dd40b49d50be3fa4cdd6c4" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb"
I20240503 02:11:49.002930 26268 raft_consensus.cc:3037] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:49.005851 26138 raft_consensus.cc:2454] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 1.
I20240503 02:11:49.006665 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495c99a8850d4ec6b52a89bec172bab0" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:11:49.007525 26268 raft_consensus.cc:2454] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 1.
I20240503 02:11:49.007589 26658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508ee4b913dd40b49d50be3fa4cdd6c4" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
I20240503 02:11:49.007921 26658 raft_consensus.cc:3037] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:49.007907 26464 leader_election.cc:302] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, 7c384005cb3041a0bd70e163473b6acb, b2835e59d40a48aea53c5dedb3bd1e9e; no voters:
I20240503 02:11:49.008212 27050 raft_consensus.cc:2781] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:49.011554 26074 leader_election.cc:302] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, 7c384005cb3041a0bd70e163473b6acb, b2835e59d40a48aea53c5dedb3bd1e9e; no voters:
W20240503 02:11:49.012073 26074 leader_election.cc:341] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:49.013353 26658 raft_consensus.cc:2454] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 1.
I20240503 02:11:49.014747 26398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508ee4b913dd40b49d50be3fa4cdd6c4" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce"
I20240503 02:11:49.015010 26398 raft_consensus.cc:3037] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:49.015012 27069 raft_consensus.cc:2781] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:49.015159 27069 raft_consensus.cc:482] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:49.015285 27069 raft_consensus.cc:3037] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:49.018584 26398 raft_consensus.cc:2454] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 1.
I20240503 02:11:49.018468 27040 consensus_queue.cc:260] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:49.018931 26007 raft_consensus.cc:2454] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 0.
I20240503 02:11:49.019402 27053 leader_election.cc:288] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 election: Requested vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675)
I20240503 02:11:49.020936 27040 ts_tablet_manager.cc:1398] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221: Time spent starting tablet: real 0.023s user 0.002s sys 0.001s
I20240503 02:11:49.021018 27039 tablet_bootstrap.cc:492] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce: No bootstrap required, opened a new log
I20240503 02:11:49.021178 27039 ts_tablet_manager.cc:1367] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce: Time spent bootstrapping tablet: real 0.128s user 0.031s sys 0.070s
I20240503 02:11:49.023664 27039 raft_consensus.cc:348] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:49.023949 27039 raft_consensus.cc:374] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:49.024665 27039 raft_consensus.cc:729] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Initialized, Role: FOLLOWER
I20240503 02:11:49.025127 27039 consensus_queue.cc:260] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:49.025561 27069 raft_consensus.cc:504] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:49.026381 27039 ts_tablet_manager.cc:1398] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:11:49.027046 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495c99a8850d4ec6b52a89bec172bab0" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb"
I20240503 02:11:49.027316 26268 raft_consensus.cc:3037] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:49.030725 26268 raft_consensus.cc:2454] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:49.032542 27050 raft_consensus.cc:686] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 LEADER]: Becoming Leader. State: Replica: b2835e59d40a48aea53c5dedb3bd1e9e, State: Running, Role: LEADER
I20240503 02:11:49.032603 26398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495c99a8850d4ec6b52a89bec172bab0" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:11:49.032786 26398 raft_consensus.cc:2454] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 0.
I20240503 02:11:49.033205 26204 leader_election.cc:302] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb, cff7b7a19fc64db5849391b1d8e10cce; no voters:
I20240503 02:11:49.033121 27050 consensus_queue.cc:237] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:49.033666 26955 raft_consensus.cc:2741] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 02:11:49.035596 27069 leader_election.cc:288] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 election: Requested vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675)
I20240503 02:11:49.036305 26528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495c99a8850d4ec6b52a89bec172bab0" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e"
I20240503 02:11:49.036510 26528 raft_consensus.cc:3037] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:49.037101 25859 catalog_manager.cc:5484] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e reported cstate change: term changed from 0 to 1, leader changed from <none> to b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197). New cstate: current_term: 1 leader_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:49.038317 26924 raft_consensus.cc:686] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Becoming Leader. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Running, Role: LEADER
I20240503 02:11:49.038832 26924 consensus_queue.cc:237] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:49.039067 26398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495c99a8850d4ec6b52a89bec172bab0" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce"
I20240503 02:11:49.039160 26528 raft_consensus.cc:2454] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:49.039244 26398 raft_consensus.cc:3037] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:49.040979 26075 leader_election.cc:302] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, 7c384005cb3041a0bd70e163473b6acb, b2835e59d40a48aea53c5dedb3bd1e9e; no voters:
I20240503 02:11:49.041991 26398 raft_consensus.cc:2454] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:49.041991 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495c99a8850d4ec6b52a89bec172bab0" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:11:49.042313 26138 raft_consensus.cc:2379] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c384005cb3041a0bd70e163473b6acb in current term 1: Already voted for candidate 2cf1cac003b44643a2d4de1204aacf0f in this term.
I20240503 02:11:49.042739 26528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495c99a8850d4ec6b52a89bec172bab0" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:11:49.042960 26528 raft_consensus.cc:2379] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c384005cb3041a0bd70e163473b6acb in current term 1: Already voted for candidate 2cf1cac003b44643a2d4de1204aacf0f in this term.
I20240503 02:11:49.045419 27069 raft_consensus.cc:2781] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:49.045663 27069 raft_consensus.cc:686] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Becoming Leader. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Running, Role: LEADER
I20240503 02:11:49.046236 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495c99a8850d4ec6b52a89bec172bab0" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221"
I20240503 02:11:49.046111 27069 consensus_queue.cc:237] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:49.046445 26008 raft_consensus.cc:3037] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:49.046402 26007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495c99a8850d4ec6b52a89bec172bab0" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:11:49.048631 25859 catalog_manager.cc:5484] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194). New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:49.049217 26008 raft_consensus.cc:2454] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:49.053244 25859 catalog_manager.cc:5484] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194). New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: HEALTHY } } }
W20240503 02:11:49.069974 25941 consensus_peers.cc:485] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.199:34279: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:11:49.382263 27075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:49.382555 27075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:49.390141 27075 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:11:49.447907 27069 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:49.457697 27069 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:49.462332 27069 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:49.468082 27069 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:49.471230 27062 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:49.507331 25941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.104.199:34279: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:11:49.531140 27050 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:49.543032 26398 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:49.544059 26138 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:49.544682 27050 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:49.545169 27050 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:49.562559 26397 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:49.563246 27062 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:49.569478 26269 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:49.569582 26008 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:49.570132 27102 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:49.573490 27102 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:49.637307 27123 mvcc.cc:205] Tried to move back new op lower bound from 7023420659961417728 to 7023420657858498560. Current Snapshot: MvccSnapshot[applied={T|T < 7023420659961417728}]
I20240503 02:11:49.648160 26268 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:11:49.686028 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 113228us
I20240503 02:11:49.692235 27121 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:49.697260 26462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 152737us
I20240503 02:11:49.713085 27101 mvcc.cc:205] Tried to move back new op lower bound from 7023420659882209280 to 7023420657805389824. Current Snapshot: MvccSnapshot[applied={T|T < 7023420659882209280}]
I20240503 02:11:49.745982 27136 mvcc.cc:205] Tried to move back new op lower bound from 7023420659882209280 to 7023420657805389824. Current Snapshot: MvccSnapshot[applied={T|T < 7023420659882209280}]
W20240503 02:11:49.752425 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 64722.8us
I20240503 02:11:49.757825 27104 mvcc.cc:205] Tried to move back new op lower bound from 7023420659961417728 to 7023420657858498560. Current Snapshot: MvccSnapshot[applied={T|T < 7023420659961417728}]
I20240503 02:11:49.758111 27116 mvcc.cc:205] Tried to move back new op lower bound from 7023420659961417728 to 7023420657858498560. Current Snapshot: MvccSnapshot[applied={T|T < 7023420659961417728}]
I20240503 02:11:49.769466 27108 mvcc.cc:205] Tried to move back new op lower bound from 7023420659882209280 to 7023420657805389824. Current Snapshot: MvccSnapshot[applied={T|T < 7023420659882209280}]
INSERT report
rows total: 2000
time total: 464.002 ms
time per row: 0.232001 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:11:50.917829 25864 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54512:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:11:50.933462 26249 tablet_service.cc:1467] Processing CreateTablet for tablet 39d3ba2cbbad40d6a232e534c65d2d74 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.933851 26118 tablet_service.cc:1467] Processing CreateTablet for tablet 39d3ba2cbbad40d6a232e534c65d2d74 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.933935 26249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d3ba2cbbad40d6a232e534c65d2d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.933903 25987 tablet_service.cc:1467] Processing CreateTablet for tablet 39d3ba2cbbad40d6a232e534c65d2d74 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.934224 26118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d3ba2cbbad40d6a232e534c65d2d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.934309 25987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d3ba2cbbad40d6a232e534c65d2d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.938045 26507 tablet_service.cc:1467] Processing CreateTablet for tablet 740218c49aa84ee28f875f2cfcee01b8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.938467 26507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740218c49aa84ee28f875f2cfcee01b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.938410 26639 tablet_service.cc:1467] Processing CreateTablet for tablet 39d3ba2cbbad40d6a232e534c65d2d74 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.938782 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d3ba2cbbad40d6a232e534c65d2d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.940611 25988 tablet_service.cc:1467] Processing CreateTablet for tablet 740218c49aa84ee28f875f2cfcee01b8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.941054 25988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740218c49aa84ee28f875f2cfcee01b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.944212 26638 tablet_service.cc:1467] Processing CreateTablet for tablet 740218c49aa84ee28f875f2cfcee01b8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.944555 26638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740218c49aa84ee28f875f2cfcee01b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.948645 26248 tablet_service.cc:1467] Processing CreateTablet for tablet 740218c49aa84ee28f875f2cfcee01b8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.948635 26378 tablet_service.cc:1467] Processing CreateTablet for tablet 39d3ba2cbbad40d6a232e534c65d2d74 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.949028 26248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740218c49aa84ee28f875f2cfcee01b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.949023 26378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d3ba2cbbad40d6a232e534c65d2d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.952850 26119 tablet_service.cc:1467] Processing CreateTablet for tablet 740218c49aa84ee28f875f2cfcee01b8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.953269 26119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740218c49aa84ee28f875f2cfcee01b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.956612 26377 tablet_service.cc:1467] Processing CreateTablet for tablet fa2fbede85374d1492edeaafef243747 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.957108 26377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa2fbede85374d1492edeaafef243747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.958122 27159 tablet_bootstrap.cc:492] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e: Bootstrap starting.
I20240503 02:11:50.961550 27159 tablet_bootstrap.cc:654] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:50.972949 25988 tablet_service.cc:1467] Processing CreateTablet for tablet fa2fbede85374d1492edeaafef243747 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.972949 26117 tablet_service.cc:1467] Processing CreateTablet for tablet fa2fbede85374d1492edeaafef243747 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.973373 26117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa2fbede85374d1492edeaafef243747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.973376 26637 tablet_service.cc:1467] Processing CreateTablet for tablet fa2fbede85374d1492edeaafef243747 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.973559 25988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa2fbede85374d1492edeaafef243747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.973819 26637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa2fbede85374d1492edeaafef243747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.976202 26507 tablet_service.cc:1467] Processing CreateTablet for tablet fa2fbede85374d1492edeaafef243747 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.976608 26507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa2fbede85374d1492edeaafef243747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.983387 27160 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce: Bootstrap starting.
I20240503 02:11:50.984337 27163 tablet_bootstrap.cc:492] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap starting.
I20240503 02:11:50.986518 27163 tablet_bootstrap.cc:654] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:50.987179 27160 tablet_bootstrap.cc:654] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:50.988602 27161 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap starting.
I20240503 02:11:50.991528 27161 tablet_bootstrap.cc:654] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:50.992993 27160 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce: No bootstrap required, opened a new log
I20240503 02:11:50.993216 27160 ts_tablet_manager.cc:1367] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20240503 02:11:50.994150 27160 raft_consensus.cc:348] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:50.994583 27160 raft_consensus.cc:374] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:50.994729 27160 raft_consensus.cc:729] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Initialized, Role: FOLLOWER
I20240503 02:11:50.995080 27160 consensus_queue.cc:260] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:50.995944 27158 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb: Bootstrap starting.
I20240503 02:11:50.996289 27160 ts_tablet_manager.cc:1398] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:11:50.996456 27164 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230: Bootstrap starting.
I20240503 02:11:50.996598 27160 tablet_bootstrap.cc:492] T fa2fbede85374d1492edeaafef243747 P cff7b7a19fc64db5849391b1d8e10cce: Bootstrap starting.
I20240503 02:11:50.997442 27158 tablet_bootstrap.cc:654] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:50.998862 27160 tablet_bootstrap.cc:654] T fa2fbede85374d1492edeaafef243747 P cff7b7a19fc64db5849391b1d8e10cce: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:50.999500 27159 tablet_bootstrap.cc:492] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e: No bootstrap required, opened a new log
I20240503 02:11:50.999727 27159 ts_tablet_manager.cc:1367] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.003s
I20240503 02:11:51.000587 27159 raft_consensus.cc:348] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:51.000886 27159 raft_consensus.cc:374] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.001037 27159 raft_consensus.cc:729] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2835e59d40a48aea53c5dedb3bd1e9e, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.001417 27159 consensus_queue.cc:260] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:51.002107 27159 ts_tablet_manager.cc:1398] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:11:51.004163 27159 tablet_bootstrap.cc:492] T fa2fbede85374d1492edeaafef243747 P b2835e59d40a48aea53c5dedb3bd1e9e: Bootstrap starting.
I20240503 02:11:51.005061 27158 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb: No bootstrap required, opened a new log
I20240503 02:11:51.005717 27158 ts_tablet_manager.cc:1367] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240503 02:11:51.006857 27161 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221: No bootstrap required, opened a new log
I20240503 02:11:51.007041 27161 ts_tablet_manager.cc:1367] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s
I20240503 02:11:51.007881 27161 raft_consensus.cc:348] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:51.008523 27161 raft_consensus.cc:374] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.008641 27164 tablet_bootstrap.cc:654] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:51.008683 27161 raft_consensus.cc:729] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.009049 27161 consensus_queue.cc:260] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:51.009162 27158 raft_consensus.cc:348] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:51.009503 27158 raft_consensus.cc:374] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.009616 27158 raft_consensus.cc:729] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.010080 27163 tablet_bootstrap.cc:492] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f: No bootstrap required, opened a new log
I20240503 02:11:51.009949 27158 consensus_queue.cc:260] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:51.010564 27159 tablet_bootstrap.cc:654] T fa2fbede85374d1492edeaafef243747 P b2835e59d40a48aea53c5dedb3bd1e9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:51.011319 27158 ts_tablet_manager.cc:1398] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:11:51.010224 27163 ts_tablet_manager.cc:1367] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.009s
I20240503 02:11:51.011636 27158 tablet_bootstrap.cc:492] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb: Bootstrap starting.
I20240503 02:11:51.012137 27161 ts_tablet_manager.cc:1398] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:11:51.012137 27163 raft_consensus.cc:348] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:51.012356 27163 raft_consensus.cc:374] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.012444 27163 raft_consensus.cc:729] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.012655 27163 consensus_queue.cc:260] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:51.013151 27158 tablet_bootstrap.cc:654] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:51.013150 27163 ts_tablet_manager.cc:1398] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:11:51.015081 27161 tablet_bootstrap.cc:492] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap starting.
I20240503 02:11:51.015221 27160 tablet_bootstrap.cc:492] T fa2fbede85374d1492edeaafef243747 P cff7b7a19fc64db5849391b1d8e10cce: No bootstrap required, opened a new log
I20240503 02:11:51.015463 27160 ts_tablet_manager.cc:1367] T fa2fbede85374d1492edeaafef243747 P cff7b7a19fc64db5849391b1d8e10cce: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.004s
I20240503 02:11:51.016130 27163 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap starting.
I20240503 02:11:51.016363 27160 raft_consensus.cc:348] T fa2fbede85374d1492edeaafef243747 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:51.016774 27160 raft_consensus.cc:374] T fa2fbede85374d1492edeaafef243747 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.016891 27160 raft_consensus.cc:729] T fa2fbede85374d1492edeaafef243747 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.017208 27160 consensus_queue.cc:260] T fa2fbede85374d1492edeaafef243747 P cff7b7a19fc64db5849391b1d8e10cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:51.018082 27160 ts_tablet_manager.cc:1398] T fa2fbede85374d1492edeaafef243747 P cff7b7a19fc64db5849391b1d8e10cce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:51.019556 27161 tablet_bootstrap.cc:654] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:51.021029 27163 tablet_bootstrap.cc:654] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:51.021229 27158 tablet_bootstrap.cc:492] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb: No bootstrap required, opened a new log
I20240503 02:11:51.021410 27158 ts_tablet_manager.cc:1367] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240503 02:11:51.022241 27158 raft_consensus.cc:348] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:51.022619 27158 raft_consensus.cc:374] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.022737 27158 raft_consensus.cc:729] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.023051 27158 consensus_queue.cc:260] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:51.023708 27158 ts_tablet_manager.cc:1398] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:11:51.027081 27159 tablet_bootstrap.cc:492] T fa2fbede85374d1492edeaafef243747 P b2835e59d40a48aea53c5dedb3bd1e9e: No bootstrap required, opened a new log
I20240503 02:11:51.027288 27159 ts_tablet_manager.cc:1367] T fa2fbede85374d1492edeaafef243747 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.004s
I20240503 02:11:51.028052 27159 raft_consensus.cc:348] T fa2fbede85374d1492edeaafef243747 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:51.028342 27159 raft_consensus.cc:374] T fa2fbede85374d1492edeaafef243747 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.028502 27159 raft_consensus.cc:729] T fa2fbede85374d1492edeaafef243747 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2835e59d40a48aea53c5dedb3bd1e9e, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.028577 27164 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230: No bootstrap required, opened a new log
I20240503 02:11:51.028790 27164 ts_tablet_manager.cc:1367] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.000s
I20240503 02:11:51.028878 27159 consensus_queue.cc:260] T fa2fbede85374d1492edeaafef243747 P b2835e59d40a48aea53c5dedb3bd1e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:51.029618 27159 ts_tablet_manager.cc:1398] T fa2fbede85374d1492edeaafef243747 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:51.029644 27164 raft_consensus.cc:348] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:51.030033 27164 raft_consensus.cc:374] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.030167 27164 raft_consensus.cc:729] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.030637 27164 consensus_queue.cc:260] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:51.031311 26986 raft_consensus.cc:482] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:51.031518 26986 raft_consensus.cc:504] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:51.032925 26986 leader_election.cc:288] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369)
I20240503 02:11:51.033490 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d3ba2cbbad40d6a232e534c65d2d74" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:11:51.033624 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d3ba2cbbad40d6a232e534c65d2d74" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:11:51.033798 26268 raft_consensus.cc:2454] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 0.
I20240503 02:11:51.033418 27164 ts_tablet_manager.cc:1398] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:11:51.034633 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d3ba2cbbad40d6a232e534c65d2d74" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
W20240503 02:11:51.034621 26332 leader_election.cc:341] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:51.034816 26008 raft_consensus.cc:2454] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 0.
I20240503 02:11:51.035317 26332 leader_election.cc:302] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb, cff7b7a19fc64db5849391b1d8e10cce; no voters: 2cf1cac003b44643a2d4de1204aacf0f
I20240503 02:11:51.035732 26986 raft_consensus.cc:2781] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:51.035863 26986 raft_consensus.cc:482] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:51.035995 26986 raft_consensus.cc:3037] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.037127 27164 tablet_bootstrap.cc:492] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230: Bootstrap starting.
I20240503 02:11:51.039335 27164 tablet_bootstrap.cc:654] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:51.039460 26986 raft_consensus.cc:504] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:51.040846 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d3ba2cbbad40d6a232e534c65d2d74" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb"
I20240503 02:11:51.040870 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d3ba2cbbad40d6a232e534c65d2d74" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f"
I20240503 02:11:51.041016 26268 raft_consensus.cc:3037] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:11:51.041388 26332 leader_election.cc:341] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:51.041663 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d3ba2cbbad40d6a232e534c65d2d74" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221"
I20240503 02:11:51.041698 26986 leader_election.cc:288] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 election: Requested vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369)
I20240503 02:11:51.041810 26008 raft_consensus.cc:3037] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.043210 27170 raft_consensus.cc:482] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:51.043329 27170 raft_consensus.cc:504] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:51.043694 26268 raft_consensus.cc:2454] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 1.
I20240503 02:11:51.044203 27170 leader_election.cc:288] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:11:51.045058 26008 raft_consensus.cc:2454] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 1.
I20240503 02:11:51.045413 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740218c49aa84ee28f875f2cfcee01b8" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:11:51.045526 26332 leader_election.cc:302] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb, cff7b7a19fc64db5849391b1d8e10cce; no voters: 2cf1cac003b44643a2d4de1204aacf0f
I20240503 02:11:51.045545 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740218c49aa84ee28f875f2cfcee01b8" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:11:51.045743 26138 raft_consensus.cc:2454] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 0.
I20240503 02:11:51.045938 26986 raft_consensus.cc:2781] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader election won for term 1
W20240503 02:11:51.046051 26202 leader_election.cc:341] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:51.056671 26528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740218c49aa84ee28f875f2cfcee01b8" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:11:51.056874 26528 raft_consensus.cc:2454] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 0.
I20240503 02:11:51.057370 26205 leader_election.cc:302] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, 7c384005cb3041a0bd70e163473b6acb, b2835e59d40a48aea53c5dedb3bd1e9e; no voters: 3bb103b2fd2344e7b55caf7389acf221
I20240503 02:11:51.057682 27170 raft_consensus.cc:2781] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:51.057770 27170 raft_consensus.cc:482] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:51.057847 27170 raft_consensus.cc:3037] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.060851 27170 raft_consensus.cc:504] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:51.062278 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740218c49aa84ee28f875f2cfcee01b8" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221"
W20240503 02:11:51.062789 26202 leader_election.cc:341] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:51.062947 26658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d3ba2cbbad40d6a232e534c65d2d74" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:11:51.063154 26658 raft_consensus.cc:2454] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 0.
I20240503 02:11:51.063612 26658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d3ba2cbbad40d6a232e534c65d2d74" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
I20240503 02:11:51.063846 26658 raft_consensus.cc:3037] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.064093 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740218c49aa84ee28f875f2cfcee01b8" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f"
I20240503 02:11:51.064254 26138 raft_consensus.cc:3037] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.066591 26658 raft_consensus.cc:2454] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 1.
I20240503 02:11:51.066933 26138 raft_consensus.cc:2454] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 1.
I20240503 02:11:51.067770 27170 leader_election.cc:288] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 election: Requested vote from peers 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:11:51.068424 26528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740218c49aa84ee28f875f2cfcee01b8" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e"
I20240503 02:11:51.068624 26528 raft_consensus.cc:3037] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.071270 26528 raft_consensus.cc:2454] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 1.
I20240503 02:11:51.071913 26986 raft_consensus.cc:686] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 1 LEADER]: Becoming Leader. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Running, Role: LEADER
I20240503 02:11:51.072255 26986 consensus_queue.cc:237] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:51.072614 26205 leader_election.cc:302] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, 7c384005cb3041a0bd70e163473b6acb, b2835e59d40a48aea53c5dedb3bd1e9e; no voters: 3bb103b2fd2344e7b55caf7389acf221
I20240503 02:11:51.072872 26658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740218c49aa84ee28f875f2cfcee01b8" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:11:51.073024 27164 tablet_bootstrap.cc:492] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230: No bootstrap required, opened a new log
I20240503 02:11:51.073210 27164 ts_tablet_manager.cc:1367] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.007s
I20240503 02:11:51.073266 27170 raft_consensus.cc:2781] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:51.073465 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740218c49aa84ee28f875f2cfcee01b8" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
W20240503 02:11:51.075492 26205 leader_election.cc:341] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:51.075716 27161 tablet_bootstrap.cc:492] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221: No bootstrap required, opened a new log
I20240503 02:11:51.075925 27161 ts_tablet_manager.cc:1367] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.002s
I20240503 02:11:51.076993 27163 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f: No bootstrap required, opened a new log
I20240503 02:11:51.074196 27164 raft_consensus.cc:348] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:51.077247 27163 ts_tablet_manager.cc:1367] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.026s
W20240503 02:11:51.077301 26205 leader_election.cc:341] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:51.077404 27164 raft_consensus.cc:374] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.077499 27164 raft_consensus.cc:729] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.076658 27161 raft_consensus.cc:348] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:51.078898 27163 raft_consensus.cc:348] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:51.079224 27161 raft_consensus.cc:374] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.079224 27163 raft_consensus.cc:374] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.079305 27163 raft_consensus.cc:729] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.079433 27161 raft_consensus.cc:729] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.079579 27163 consensus_queue.cc:260] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:51.078227 27164 consensus_queue.cc:260] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:51.080094 27161 consensus_queue.cc:260] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:51.080047 25858 catalog_manager.cc:5484] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: term changed from 0 to 1, leader changed from <none> to cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196). New cstate: current_term: 1 leader_uuid: "cff7b7a19fc64db5849391b1d8e10cce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:51.080797 27163 ts_tablet_manager.cc:1398] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:11:51.081089 27163 tablet_bootstrap.cc:492] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap starting.
I20240503 02:11:51.082521 27164 ts_tablet_manager.cc:1398] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240503 02:11:51.082677 27163 tablet_bootstrap.cc:654] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:51.082825 27164 tablet_bootstrap.cc:492] T fa2fbede85374d1492edeaafef243747 P ae8b30bcfa5b425782a95374787dd230: Bootstrap starting.
I20240503 02:11:51.085377 27164 tablet_bootstrap.cc:654] T fa2fbede85374d1492edeaafef243747 P ae8b30bcfa5b425782a95374787dd230: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:51.086081 27161 ts_tablet_manager.cc:1398] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 02:11:51.088352 27161 tablet_bootstrap.cc:492] T fa2fbede85374d1492edeaafef243747 P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap starting.
I20240503 02:11:51.090334 27170 raft_consensus.cc:686] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Becoming Leader. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Running, Role: LEADER
I20240503 02:11:51.090994 27170 consensus_queue.cc:237] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:51.091286 27164 tablet_bootstrap.cc:492] T fa2fbede85374d1492edeaafef243747 P ae8b30bcfa5b425782a95374787dd230: No bootstrap required, opened a new log
I20240503 02:11:51.091415 27164 ts_tablet_manager.cc:1367] T fa2fbede85374d1492edeaafef243747 P ae8b30bcfa5b425782a95374787dd230: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240503 02:11:51.092159 27161 tablet_bootstrap.cc:654] T fa2fbede85374d1492edeaafef243747 P 3bb103b2fd2344e7b55caf7389acf221: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:51.092154 27164 raft_consensus.cc:348] T fa2fbede85374d1492edeaafef243747 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:51.092459 27164 raft_consensus.cc:374] T fa2fbede85374d1492edeaafef243747 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.092602 27164 raft_consensus.cc:729] T fa2fbede85374d1492edeaafef243747 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.092954 27163 tablet_bootstrap.cc:492] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f: No bootstrap required, opened a new log
I20240503 02:11:51.092890 27164 consensus_queue.cc:260] T fa2fbede85374d1492edeaafef243747 P ae8b30bcfa5b425782a95374787dd230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:51.093194 27163 ts_tablet_manager.cc:1367] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20240503 02:11:51.094179 27163 raft_consensus.cc:348] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:51.094470 27163 raft_consensus.cc:374] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.094602 27163 raft_consensus.cc:729] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.094674 27164 ts_tablet_manager.cc:1398] T fa2fbede85374d1492edeaafef243747 P ae8b30bcfa5b425782a95374787dd230: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:51.094878 27163 consensus_queue.cc:260] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:51.096411 27163 ts_tablet_manager.cc:1398] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:11:51.097252 25864 catalog_manager.cc:5484] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c384005cb3041a0bd70e163473b6acb (127.23.104.195). New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:51.097707 27161 tablet_bootstrap.cc:492] T fa2fbede85374d1492edeaafef243747 P 3bb103b2fd2344e7b55caf7389acf221: No bootstrap required, opened a new log
I20240503 02:11:51.097865 27161 ts_tablet_manager.cc:1367] T fa2fbede85374d1492edeaafef243747 P 3bb103b2fd2344e7b55caf7389acf221: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20240503 02:11:51.098608 27161 raft_consensus.cc:348] T fa2fbede85374d1492edeaafef243747 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:51.098862 27161 raft_consensus.cc:374] T fa2fbede85374d1492edeaafef243747 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.098985 27161 raft_consensus.cc:729] T fa2fbede85374d1492edeaafef243747 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.099244 27161 consensus_queue.cc:260] T fa2fbede85374d1492edeaafef243747 P 3bb103b2fd2344e7b55caf7389acf221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:51.100461 27161 ts_tablet_manager.cc:1398] T fa2fbede85374d1492edeaafef243747 P 3bb103b2fd2344e7b55caf7389acf221: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:11:51.101151 27062 raft_consensus.cc:482] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:51.101290 27062 raft_consensus.cc:504] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:51.102115 27062 leader_election.cc:288] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369)
I20240503 02:11:51.102514 26397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d3ba2cbbad40d6a232e534c65d2d74" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:11:51.102532 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d3ba2cbbad40d6a232e534c65d2d74" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:11:51.102597 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d3ba2cbbad40d6a232e534c65d2d74" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:11:51.102849 26268 raft_consensus.cc:2379] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2cf1cac003b44643a2d4de1204aacf0f in current term 1: Already voted for candidate cff7b7a19fc64db5849391b1d8e10cce in this term.
I20240503 02:11:51.102878 26659 raft_consensus.cc:2379] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2cf1cac003b44643a2d4de1204aacf0f in current term 1: Already voted for candidate cff7b7a19fc64db5849391b1d8e10cce in this term.
I20240503 02:11:51.103163 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d3ba2cbbad40d6a232e534c65d2d74" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:11:51.103413 26008 raft_consensus.cc:2379] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2cf1cac003b44643a2d4de1204aacf0f in current term 1: Already voted for candidate cff7b7a19fc64db5849391b1d8e10cce in this term.
I20240503 02:11:51.103917 26074 leader_election.cc:302] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f; no voters: 7c384005cb3041a0bd70e163473b6acb, ae8b30bcfa5b425782a95374787dd230, cff7b7a19fc64db5849391b1d8e10cce
I20240503 02:11:51.104251 27062 raft_consensus.cc:3037] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.106487 27062 raft_consensus.cc:2726] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:11:51.138429 27062 raft_consensus.cc:482] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:51.138638 27062 raft_consensus.cc:504] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:51.139986 27062 leader_election.cc:288] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347)
I20240503 02:11:51.139927 26397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2fbede85374d1492edeaafef243747" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:11:51.140014 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2fbede85374d1492edeaafef243747" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:11:51.140202 26397 raft_consensus.cc:2454] T fa2fbede85374d1492edeaafef243747 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 0.
I20240503 02:11:51.140275 26659 raft_consensus.cc:2454] T fa2fbede85374d1492edeaafef243747 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 0.
I20240503 02:11:51.140393 26528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2fbede85374d1492edeaafef243747" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:11:51.140645 26528 raft_consensus.cc:2454] T fa2fbede85374d1492edeaafef243747 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 0.
I20240503 02:11:51.140722 26075 leader_election.cc:302] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, ae8b30bcfa5b425782a95374787dd230, cff7b7a19fc64db5849391b1d8e10cce; no voters:
I20240503 02:11:51.140823 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2fbede85374d1492edeaafef243747" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:11:51.141080 26008 raft_consensus.cc:2454] T fa2fbede85374d1492edeaafef243747 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 0.
I20240503 02:11:51.141163 27062 raft_consensus.cc:2781] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:51.141248 27062 raft_consensus.cc:482] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:51.141407 27062 raft_consensus.cc:3037] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.143661 27062 raft_consensus.cc:504] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:51.144598 27062 leader_election.cc:288] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 election: Requested vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347)
I20240503 02:11:51.144914 26528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2fbede85374d1492edeaafef243747" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e"
I20240503 02:11:51.145134 26528 raft_consensus.cc:3037] T fa2fbede85374d1492edeaafef243747 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.145092 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2fbede85374d1492edeaafef243747" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
I20240503 02:11:51.145277 26659 raft_consensus.cc:3037] T fa2fbede85374d1492edeaafef243747 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.146930 26528 raft_consensus.cc:2454] T fa2fbede85374d1492edeaafef243747 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:51.147302 26397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2fbede85374d1492edeaafef243747" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce"
I20240503 02:11:51.147527 26397 raft_consensus.cc:3037] T fa2fbede85374d1492edeaafef243747 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.147804 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2fbede85374d1492edeaafef243747" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221"
I20240503 02:11:51.147960 26659 raft_consensus.cc:2454] T fa2fbede85374d1492edeaafef243747 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:51.148005 26008 raft_consensus.cc:3037] T fa2fbede85374d1492edeaafef243747 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.148324 26075 leader_election.cc:302] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, ae8b30bcfa5b425782a95374787dd230, b2835e59d40a48aea53c5dedb3bd1e9e; no voters:
I20240503 02:11:51.148635 27062 raft_consensus.cc:2781] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:51.148869 27062 raft_consensus.cc:686] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Becoming Leader. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Running, Role: LEADER
I20240503 02:11:51.149125 27062 consensus_queue.cc:237] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:51.150327 26397 raft_consensus.cc:2454] T fa2fbede85374d1492edeaafef243747 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:51.151202 26008 raft_consensus.cc:2454] T fa2fbede85374d1492edeaafef243747 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:51.152791 25864 catalog_manager.cc:5484] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194). New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:51.415468 26867 raft_consensus.cc:482] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:51.415663 26867 raft_consensus.cc:504] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:51.416644 26867 leader_election.cc:288] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:11:51.416909 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740218c49aa84ee28f875f2cfcee01b8" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:11:51.417161 26659 raft_consensus.cc:2454] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 0.
I20240503 02:11:51.417120 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740218c49aa84ee28f875f2cfcee01b8" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:11:51.417294 26528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740218c49aa84ee28f875f2cfcee01b8" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:11:51.417523 26528 raft_consensus.cc:2379] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3bb103b2fd2344e7b55caf7389acf221 in current term 1: Already voted for candidate 7c384005cb3041a0bd70e163473b6acb in this term.
I20240503 02:11:51.424290 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740218c49aa84ee28f875f2cfcee01b8" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:11:51.424588 26138 raft_consensus.cc:2379] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3bb103b2fd2344e7b55caf7389acf221 in current term 1: Already voted for candidate 7c384005cb3041a0bd70e163473b6acb in this term.
I20240503 02:11:51.424981 25941 leader_election.cc:302] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, ae8b30bcfa5b425782a95374787dd230; no voters: 2cf1cac003b44643a2d4de1204aacf0f, 7c384005cb3041a0bd70e163473b6acb, b2835e59d40a48aea53c5dedb3bd1e9e
I20240503 02:11:51.425277 26867 raft_consensus.cc:3037] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.427141 26867 raft_consensus.cc:2726] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:11:51.470206 26659 raft_consensus.cc:3037] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.474004 27170 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:51.517787 27191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:51.518115 27191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:51.525854 27191 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:11:51.529031 26986 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:51.532020 27062 consensus_queue.cc:1035] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:51.544579 26986 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:11:51.551021 26986 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 02:11:51.585800 27215 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:51.586620 27062 consensus_queue.cc:1035] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:51.591862 27062 consensus_queue.cc:1035] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:51.603060 27062 consensus_queue.cc:1035] T fa2fbede85374d1492edeaafef243747 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:51.606029 27170 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:51.639462 27170 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:51.640022 27170 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:51.650534 25941 consensus_peers.cc:485] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.199:34279: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:11:51.871711 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 44739.1us
INSERT report
rows total: 2000
time total: 393.622 ms
time per row: 0.196811 ms
I20240503 02:11:52.152073 25858 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54512:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:11:52.161857 25987 tablet_service.cc:1467] Processing CreateTablet for tablet e866b4f1b338419ab1025287b2aaac5e (DEFAULT_TABLE table=rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.162240 26507 tablet_service.cc:1467] Processing CreateTablet for tablet e866b4f1b338419ab1025287b2aaac5e (DEFAULT_TABLE table=rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.162354 25987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e866b4f1b338419ab1025287b2aaac5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.162644 26507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e866b4f1b338419ab1025287b2aaac5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.163722 26249 tablet_service.cc:1467] Processing CreateTablet for tablet e866b4f1b338419ab1025287b2aaac5e (DEFAULT_TABLE table=rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.164170 26249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e866b4f1b338419ab1025287b2aaac5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.166872 26248 tablet_service.cc:1467] Processing CreateTablet for tablet c27502976d474b938cce2235e5784f79 (DEFAULT_TABLE table=rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.167234 26248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c27502976d474b938cce2235e5784f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.168316 26378 tablet_service.cc:1467] Processing CreateTablet for tablet e866b4f1b338419ab1025287b2aaac5e (DEFAULT_TABLE table=rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.168525 26509 tablet_service.cc:1467] Processing CreateTablet for tablet c27502976d474b938cce2235e5784f79 (DEFAULT_TABLE table=rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.168754 26378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e866b4f1b338419ab1025287b2aaac5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.168964 26509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c27502976d474b938cce2235e5784f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.172785 26639 tablet_service.cc:1467] Processing CreateTablet for tablet e866b4f1b338419ab1025287b2aaac5e (DEFAULT_TABLE table=rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.173259 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e866b4f1b338419ab1025287b2aaac5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.174098 26637 tablet_service.cc:1467] Processing CreateTablet for tablet c27502976d474b938cce2235e5784f79 (DEFAULT_TABLE table=rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.174489 26637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c27502976d474b938cce2235e5784f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.176914 25988 tablet_service.cc:1467] Processing CreateTablet for tablet c27502976d474b938cce2235e5784f79 (DEFAULT_TABLE table=rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.177348 25988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c27502976d474b938cce2235e5784f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.187444 27287 tablet_bootstrap.cc:492] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230: Bootstrap starting.
I20240503 02:11:52.190692 27285 tablet_bootstrap.cc:492] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb: Bootstrap starting.
I20240503 02:11:52.191366 27287 tablet_bootstrap.cc:654] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.191991 26377 tablet_service.cc:1467] Processing CreateTablet for tablet c27502976d474b938cce2235e5784f79 (DEFAULT_TABLE table=rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.192416 26377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c27502976d474b938cce2235e5784f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.192531 27285 tablet_bootstrap.cc:654] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.192849 26379 tablet_service.cc:1467] Processing CreateTablet for tablet c12cc2e0fb09431e95baae55803f2184 (DEFAULT_TABLE table=rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.193181 26379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12cc2e0fb09431e95baae55803f2184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.201651 26117 tablet_service.cc:1467] Processing CreateTablet for tablet c12cc2e0fb09431e95baae55803f2184 (DEFAULT_TABLE table=rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.202083 26117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12cc2e0fb09431e95baae55803f2184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.202072 25987 tablet_service.cc:1467] Processing CreateTablet for tablet c12cc2e0fb09431e95baae55803f2184 (DEFAULT_TABLE table=rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.202131 26247 tablet_service.cc:1467] Processing CreateTablet for tablet c12cc2e0fb09431e95baae55803f2184 (DEFAULT_TABLE table=rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.202481 25987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12cc2e0fb09431e95baae55803f2184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.202616 26247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12cc2e0fb09431e95baae55803f2184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.203894 27286 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap starting.
I20240503 02:11:52.205631 27286 tablet_bootstrap.cc:654] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.210094 27285 tablet_bootstrap.cc:492] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb: No bootstrap required, opened a new log
I20240503 02:11:52.210322 27285 ts_tablet_manager.cc:1367] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.004s
I20240503 02:11:52.210743 27292 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap starting.
I20240503 02:11:52.210788 26509 tablet_service.cc:1467] Processing CreateTablet for tablet c12cc2e0fb09431e95baae55803f2184 (DEFAULT_TABLE table=rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.211181 27285 raft_consensus.cc:348] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:52.211390 26509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12cc2e0fb09431e95baae55803f2184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.211479 27285 raft_consensus.cc:374] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.211580 27285 raft_consensus.cc:729] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.211820 27285 consensus_queue.cc:260] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:52.212499 27285 ts_tablet_manager.cc:1398] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:52.212828 27285 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb: Bootstrap starting.
I20240503 02:11:52.214438 27285 tablet_bootstrap.cc:654] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.214936 27292 tablet_bootstrap.cc:654] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.224063 27285 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb: No bootstrap required, opened a new log
I20240503 02:11:52.224273 27285 ts_tablet_manager.cc:1367] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 02:11:52.224437 27292 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f: No bootstrap required, opened a new log
I20240503 02:11:52.224642 27292 ts_tablet_manager.cc:1367] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20240503 02:11:52.225275 27285 raft_consensus.cc:348] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } }
I20240503 02:11:52.225798 27285 raft_consensus.cc:374] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.225692 27292 raft_consensus.cc:348] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } }
I20240503 02:11:52.225942 27285 raft_consensus.cc:729] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.226091 27292 raft_consensus.cc:374] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.226253 27292 raft_consensus.cc:729] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.226269 27285 consensus_queue.cc:260] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } }
I20240503 02:11:52.227152 27285 ts_tablet_manager.cc:1398] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:52.227530 27285 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb: Bootstrap starting.
I20240503 02:11:52.227815 27288 tablet_bootstrap.cc:492] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce: Bootstrap starting.
I20240503 02:11:52.228138 27292 consensus_queue.cc:260] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } }
I20240503 02:11:52.229103 27285 tablet_bootstrap.cc:654] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.229662 27288 tablet_bootstrap.cc:654] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.231917 27292 ts_tablet_manager.cc:1398] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:11:52.234439 27286 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221: No bootstrap required, opened a new log
I20240503 02:11:52.234858 27286 ts_tablet_manager.cc:1367] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.000s
I20240503 02:11:52.236042 27286 raft_consensus.cc:348] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } }
I20240503 02:11:52.236801 27286 raft_consensus.cc:374] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.237007 27286 raft_consensus.cc:729] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.237525 27287 tablet_bootstrap.cc:492] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230: No bootstrap required, opened a new log
I20240503 02:11:52.237432 27286 consensus_queue.cc:260] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } }
I20240503 02:11:52.237802 27287 ts_tablet_manager.cc:1367] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.000s
I20240503 02:11:52.238847 27287 raft_consensus.cc:348] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:52.239256 27286 ts_tablet_manager.cc:1398] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:11:52.239739 27286 tablet_bootstrap.cc:492] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap starting.
I20240503 02:11:52.241624 27287 raft_consensus.cc:374] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.241947 27286 tablet_bootstrap.cc:654] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.241952 27287 raft_consensus.cc:729] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.242491 27287 consensus_queue.cc:260] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:52.243145 27285 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb: No bootstrap required, opened a new log
I20240503 02:11:52.243243 27284 tablet_bootstrap.cc:492] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e: Bootstrap starting.
I20240503 02:11:52.243343 27285 ts_tablet_manager.cc:1367] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20240503 02:11:52.243973 27288 tablet_bootstrap.cc:492] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce: No bootstrap required, opened a new log
I20240503 02:11:52.244172 27288 ts_tablet_manager.cc:1367] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20240503 02:11:52.244201 27285 raft_consensus.cc:348] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } }
I20240503 02:11:52.244534 27285 raft_consensus.cc:374] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.244663 27285 raft_consensus.cc:729] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.245012 27288 raft_consensus.cc:348] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:52.245278 27288 raft_consensus.cc:374] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.245411 27288 raft_consensus.cc:729] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.244969 27285 consensus_queue.cc:260] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } }
I20240503 02:11:52.245721 27288 consensus_queue.cc:260] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:52.246452 27287 ts_tablet_manager.cc:1398] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:11:52.246665 27285 ts_tablet_manager.cc:1398] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:52.246755 27287 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230: Bootstrap starting.
I20240503 02:11:52.247551 27288 ts_tablet_manager.cc:1398] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:52.247882 27288 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce: Bootstrap starting.
I20240503 02:11:52.248435 27287 tablet_bootstrap.cc:654] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.249735 27284 tablet_bootstrap.cc:654] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.252074 27288 tablet_bootstrap.cc:654] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.254285 27286 tablet_bootstrap.cc:492] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221: No bootstrap required, opened a new log
I20240503 02:11:52.254498 27286 ts_tablet_manager.cc:1367] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20240503 02:11:52.255241 27286 raft_consensus.cc:348] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:52.255440 27286 raft_consensus.cc:374] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.255524 27286 raft_consensus.cc:729] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.256381 27287 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230: No bootstrap required, opened a new log
I20240503 02:11:52.256596 27287 ts_tablet_manager.cc:1367] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240503 02:11:52.255769 27286 consensus_queue.cc:260] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:52.257242 27288 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce: No bootstrap required, opened a new log
I20240503 02:11:52.257392 27288 ts_tablet_manager.cc:1367] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:11:52.257386 27287 raft_consensus.cc:348] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } }
I20240503 02:11:52.257678 27287 raft_consensus.cc:374] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.257767 27287 raft_consensus.cc:729] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.258023 27287 consensus_queue.cc:260] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } }
I20240503 02:11:52.258502 27286 ts_tablet_manager.cc:1398] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:11:52.258659 27287 ts_tablet_manager.cc:1398] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:11:52.258776 27286 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap starting.
I20240503 02:11:52.259171 27288 raft_consensus.cc:348] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } }
I20240503 02:11:52.259521 27288 raft_consensus.cc:374] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.259644 27288 raft_consensus.cc:729] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.259927 27288 consensus_queue.cc:260] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } }
I20240503 02:11:52.260494 27288 ts_tablet_manager.cc:1398] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:11:52.260856 27288 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce: Bootstrap starting.
I20240503 02:11:52.261034 27286 tablet_bootstrap.cc:654] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.261090 27284 tablet_bootstrap.cc:492] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e: No bootstrap required, opened a new log
I20240503 02:11:52.261267 27284 ts_tablet_manager.cc:1367] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240503 02:11:52.262200 27284 raft_consensus.cc:348] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:52.262523 27284 raft_consensus.cc:374] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.262571 27288 tablet_bootstrap.cc:654] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.262651 27284 raft_consensus.cc:729] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2835e59d40a48aea53c5dedb3bd1e9e, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.262892 27284 consensus_queue.cc:260] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:52.263687 27284 ts_tablet_manager.cc:1398] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:52.264209 27284 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e: Bootstrap starting.
I20240503 02:11:52.266597 27284 tablet_bootstrap.cc:654] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.270126 27286 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221: No bootstrap required, opened a new log
I20240503 02:11:52.270330 27286 ts_tablet_manager.cc:1367] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20240503 02:11:52.271711 27286 raft_consensus.cc:348] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } }
I20240503 02:11:52.272127 27286 raft_consensus.cc:374] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.272264 27286 raft_consensus.cc:729] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.272612 27286 consensus_queue.cc:260] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } }
I20240503 02:11:52.273311 27286 ts_tablet_manager.cc:1398] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:11:52.274549 27284 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e: No bootstrap required, opened a new log
I20240503 02:11:52.274699 27284 ts_tablet_manager.cc:1367] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240503 02:11:52.275592 27284 raft_consensus.cc:348] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } }
I20240503 02:11:52.275919 27284 raft_consensus.cc:374] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.276000 27284 raft_consensus.cc:729] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2835e59d40a48aea53c5dedb3bd1e9e, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.276229 27284 consensus_queue.cc:260] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } }
I20240503 02:11:52.276736 27284 ts_tablet_manager.cc:1398] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:11:52.276988 27284 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e: Bootstrap starting.
I20240503 02:11:52.278409 27284 tablet_bootstrap.cc:654] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.281672 27284 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e: No bootstrap required, opened a new log
I20240503 02:11:52.281740 27288 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce: No bootstrap required, opened a new log
I20240503 02:11:52.281792 27284 ts_tablet_manager.cc:1367] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:11:52.281898 27288 ts_tablet_manager.cc:1367] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.010s
I20240503 02:11:52.282328 27284 raft_consensus.cc:348] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } }
I20240503 02:11:52.282491 27284 raft_consensus.cc:374] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.282559 27284 raft_consensus.cc:729] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2835e59d40a48aea53c5dedb3bd1e9e, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.282420 27288 raft_consensus.cc:348] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } }
I20240503 02:11:52.282642 27288 raft_consensus.cc:374] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.282738 27288 raft_consensus.cc:729] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.282757 27284 consensus_queue.cc:260] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } }
I20240503 02:11:52.282951 27288 consensus_queue.cc:260] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } }
I20240503 02:11:52.283248 27284 ts_tablet_manager.cc:1398] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:11:52.283484 27288 ts_tablet_manager.cc:1398] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:11:52.329581 27222 raft_consensus.cc:482] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:52.329731 27222 raft_consensus.cc:504] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } }
I20240503 02:11:52.330549 27222 leader_election.cc:288] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999)
I20240503 02:11:52.330746 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12cc2e0fb09431e95baae55803f2184" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:11:52.330816 26397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12cc2e0fb09431e95baae55803f2184" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:11:52.330974 26008 raft_consensus.cc:2454] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 0.
I20240503 02:11:52.331036 26397 raft_consensus.cc:2454] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 0.
I20240503 02:11:52.330977 26529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12cc2e0fb09431e95baae55803f2184" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:11:52.331179 26529 raft_consensus.cc:2454] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 0.
I20240503 02:11:52.331154 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12cc2e0fb09431e95baae55803f2184" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:11:52.331354 26268 raft_consensus.cc:2454] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 0.
I20240503 02:11:52.331476 26074 leader_election.cc:302] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, 3bb103b2fd2344e7b55caf7389acf221, cff7b7a19fc64db5849391b1d8e10cce; no voters:
I20240503 02:11:52.331746 27222 raft_consensus.cc:2781] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:52.331908 27222 raft_consensus.cc:482] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:52.332000 27222 raft_consensus.cc:3037] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.333799 27222 raft_consensus.cc:504] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } }
I20240503 02:11:52.334579 27222 leader_election.cc:288] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 election: Requested vote from peers 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999)
I20240503 02:11:52.334753 26397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12cc2e0fb09431e95baae55803f2184" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce"
I20240503 02:11:52.334733 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12cc2e0fb09431e95baae55803f2184" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221"
I20240503 02:11:52.334854 26529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12cc2e0fb09431e95baae55803f2184" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e"
I20240503 02:11:52.334935 26397 raft_consensus.cc:3037] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.334918 26008 raft_consensus.cc:3037] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.335018 26529 raft_consensus.cc:3037] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.336920 26529 raft_consensus.cc:2454] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:52.337440 26397 raft_consensus.cc:2454] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:52.337404 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12cc2e0fb09431e95baae55803f2184" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb"
I20240503 02:11:52.337607 26268 raft_consensus.cc:3037] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.337841 26074 leader_election.cc:302] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, b2835e59d40a48aea53c5dedb3bd1e9e, cff7b7a19fc64db5849391b1d8e10cce; no voters:
I20240503 02:11:52.338101 27222 raft_consensus.cc:2781] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:52.338250 27222 raft_consensus.cc:686] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Becoming Leader. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Running, Role: LEADER
I20240503 02:11:52.338276 26008 raft_consensus.cc:2454] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:52.338572 27222 consensus_queue.cc:237] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } }
I20240503 02:11:52.340147 26268 raft_consensus.cc:2454] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 1.
I20240503 02:11:52.341197 25861 catalog_manager.cc:5484] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194). New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:52.434815 27235 raft_consensus.cc:482] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:52.434970 27235 raft_consensus.cc:504] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:52.436021 27235 leader_election.cc:288] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347)
I20240503 02:11:52.436635 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e866b4f1b338419ab1025287b2aaac5e" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:11:52.436892 26268 raft_consensus.cc:2454] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 0.
I20240503 02:11:52.436832 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e866b4f1b338419ab1025287b2aaac5e" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:11:52.436903 26529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e866b4f1b338419ab1025287b2aaac5e" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:11:52.437070 26659 raft_consensus.cc:2454] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 0.
I20240503 02:11:52.437124 26529 raft_consensus.cc:2454] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 0.
I20240503 02:11:52.437436 25944 leader_election.cc:302] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb, ae8b30bcfa5b425782a95374787dd230; no voters:
I20240503 02:11:52.437683 27235 raft_consensus.cc:2781] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:52.437824 27235 raft_consensus.cc:482] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:52.437908 27235 raft_consensus.cc:3037] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.440486 27235 raft_consensus.cc:504] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:52.441274 27235 leader_election.cc:288] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 1 election: Requested vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347)
I20240503 02:11:52.441520 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e866b4f1b338419ab1025287b2aaac5e" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb"
I20240503 02:11:52.441695 26268 raft_consensus.cc:3037] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.441691 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e866b4f1b338419ab1025287b2aaac5e" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
I20240503 02:11:52.441825 26529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e866b4f1b338419ab1025287b2aaac5e" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e"
I20240503 02:11:52.441871 26659 raft_consensus.cc:3037] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.441987 26529 raft_consensus.cc:3037] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.443614 26399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e866b4f1b338419ab1025287b2aaac5e" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce"
I20240503 02:11:52.443848 26399 raft_consensus.cc:3037] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.444339 26659 raft_consensus.cc:2454] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 1.
I20240503 02:11:52.444365 26268 raft_consensus.cc:2454] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 1.
I20240503 02:11:52.444500 26529 raft_consensus.cc:2454] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 1.
I20240503 02:11:52.444686 26397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e866b4f1b338419ab1025287b2aaac5e" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:11:52.444957 25944 leader_election.cc:302] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb, ae8b30bcfa5b425782a95374787dd230; no voters:
I20240503 02:11:52.445281 27235 raft_consensus.cc:2781] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:52.445721 27235 raft_consensus.cc:686] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [term 1 LEADER]: Becoming Leader. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Running, Role: LEADER
I20240503 02:11:52.446034 27235 consensus_queue.cc:237] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } }
I20240503 02:11:52.446373 26399 raft_consensus.cc:2454] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 1.
I20240503 02:11:52.449101 25861 catalog_manager.cc:5484] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "3bb103b2fd2344e7b55caf7389acf221" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:52.513693 27231 raft_consensus.cc:482] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:52.513860 27231 raft_consensus.cc:504] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } }
I20240503 02:11:52.514847 27231 leader_election.cc:288] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675)
I20240503 02:11:52.515347 26529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27502976d474b938cce2235e5784f79" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:11:52.515363 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27502976d474b938cce2235e5784f79" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:11:52.515391 26399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27502976d474b938cce2235e5784f79" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:11:52.515388 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27502976d474b938cce2235e5784f79" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:11:52.515568 26529 raft_consensus.cc:2454] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 0.
I20240503 02:11:52.515578 26008 raft_consensus.cc:2454] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 0.
I20240503 02:11:52.515604 26399 raft_consensus.cc:2454] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 0.
I20240503 02:11:52.515604 26659 raft_consensus.cc:2454] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 0.
I20240503 02:11:52.516029 26202 leader_election.cc:302] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb, b2835e59d40a48aea53c5dedb3bd1e9e; no voters:
I20240503 02:11:52.516290 27231 raft_consensus.cc:2781] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:52.516391 27231 raft_consensus.cc:482] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:52.516481 27231 raft_consensus.cc:3037] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.518213 27231 raft_consensus.cc:504] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } }
I20240503 02:11:52.519274 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27502976d474b938cce2235e5784f79" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221"
I20240503 02:11:52.519321 26529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27502976d474b938cce2235e5784f79" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e"
I20240503 02:11:52.519344 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27502976d474b938cce2235e5784f79" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
I20240503 02:11:52.519414 26399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27502976d474b938cce2235e5784f79" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce"
I20240503 02:11:52.519479 26529 raft_consensus.cc:3037] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.519492 26008 raft_consensus.cc:3037] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.519541 26659 raft_consensus.cc:3037] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.519569 26399 raft_consensus.cc:3037] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.521284 27231 leader_election.cc:288] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 election: Requested vote from peers ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675)
I20240503 02:11:52.522756 26529 raft_consensus.cc:2454] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 1.
I20240503 02:11:52.522789 26008 raft_consensus.cc:2454] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 1.
I20240503 02:11:52.522815 26659 raft_consensus.cc:2454] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 1.
I20240503 02:11:52.523000 26399 raft_consensus.cc:2454] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 1.
I20240503 02:11:52.523226 26205 leader_election.cc:302] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb, ae8b30bcfa5b425782a95374787dd230; no voters:
I20240503 02:11:52.523499 27231 raft_consensus.cc:2781] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:52.523828 27231 raft_consensus.cc:686] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Becoming Leader. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Running, Role: LEADER
I20240503 02:11:52.524104 27231 consensus_queue.cc:237] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } }
I20240503 02:11:52.526949 25861 catalog_manager.cc:5484] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c384005cb3041a0bd70e163473b6acb (127.23.104.195). New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:52.833892 27235 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:52.868221 27235 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:52.874029 27322 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:52.874760 27314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:52.875116 27314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:11:52.876194 27235 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:52.888300 27222 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:52.889706 27314 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:11:52.899736 27222 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:52.909723 27222 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:52.914340 27222 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240503 02:11:52.944921 27231 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:52.954213 27231 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:52.996577 27312 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:52.998598 27312 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 425.615 ms
time per row: 0.212808 ms
W20240503 02:11:53.399259 25941 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 59154us
I20240503 02:11:53.497421 25859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54512:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:11:53.506552 26639 tablet_service.cc:1467] Processing CreateTablet for tablet d972dd11fcd9484f9a59f63d89f1e6e2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.506695 26247 tablet_service.cc:1467] Processing CreateTablet for tablet d972dd11fcd9484f9a59f63d89f1e6e2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.506960 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d972dd11fcd9484f9a59f63d89f1e6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.507123 26247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d972dd11fcd9484f9a59f63d89f1e6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.507674 26249 tablet_service.cc:1467] Processing CreateTablet for tablet 21e7ddb56be1499b94920ebd252595ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.508119 26249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e7ddb56be1499b94920ebd252595ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.511365 26377 tablet_service.cc:1467] Processing CreateTablet for tablet 1162f9075da248d0a919db2eb45aa949 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.511492 26248 tablet_service.cc:1467] Processing CreateTablet for tablet 1162f9075da248d0a919db2eb45aa949 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.511758 26377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1162f9075da248d0a919db2eb45aa949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.511906 26248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1162f9075da248d0a919db2eb45aa949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.512207 26378 tablet_service.cc:1467] Processing CreateTablet for tablet d972dd11fcd9484f9a59f63d89f1e6e2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.512602 26378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d972dd11fcd9484f9a59f63d89f1e6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.514657 26507 tablet_service.cc:1467] Processing CreateTablet for tablet 21e7ddb56be1499b94920ebd252595ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.515064 26507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e7ddb56be1499b94920ebd252595ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.517499 26637 tablet_service.cc:1467] Processing CreateTablet for tablet 21e7ddb56be1499b94920ebd252595ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.517629 26117 tablet_service.cc:1467] Processing CreateTablet for tablet d972dd11fcd9484f9a59f63d89f1e6e2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.517874 26637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e7ddb56be1499b94920ebd252595ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.517994 26117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d972dd11fcd9484f9a59f63d89f1e6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.518599 26638 tablet_service.cc:1467] Processing CreateTablet for tablet 1162f9075da248d0a919db2eb45aa949 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.518963 26638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1162f9075da248d0a919db2eb45aa949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.520133 26119 tablet_service.cc:1467] Processing CreateTablet for tablet 21e7ddb56be1499b94920ebd252595ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.520247 26509 tablet_service.cc:1467] Processing CreateTablet for tablet 1162f9075da248d0a919db2eb45aa949 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.520502 26119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e7ddb56be1499b94920ebd252595ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.520697 26509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1162f9075da248d0a919db2eb45aa949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.524108 26118 tablet_service.cc:1467] Processing CreateTablet for tablet 1162f9075da248d0a919db2eb45aa949 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.524284 25987 tablet_service.cc:1467] Processing CreateTablet for tablet 21e7ddb56be1499b94920ebd252595ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.524458 26118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1162f9075da248d0a919db2eb45aa949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.524794 25987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e7ddb56be1499b94920ebd252595ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.526237 27404 tablet_bootstrap.cc:492] T 1162f9075da248d0a919db2eb45aa949 P cff7b7a19fc64db5849391b1d8e10cce: Bootstrap starting.
I20240503 02:11:53.527863 27404 tablet_bootstrap.cc:654] T 1162f9075da248d0a919db2eb45aa949 P cff7b7a19fc64db5849391b1d8e10cce: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.515697 25988 tablet_service.cc:1467] Processing CreateTablet for tablet d972dd11fcd9484f9a59f63d89f1e6e2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.528371 25988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d972dd11fcd9484f9a59f63d89f1e6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.535140 27407 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap starting.
I20240503 02:11:53.536406 27406 tablet_bootstrap.cc:492] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e: Bootstrap starting.
I20240503 02:11:53.538741 27407 tablet_bootstrap.cc:654] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.540966 27406 tablet_bootstrap.cc:654] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.556968 27404 tablet_bootstrap.cc:492] T 1162f9075da248d0a919db2eb45aa949 P cff7b7a19fc64db5849391b1d8e10cce: No bootstrap required, opened a new log
I20240503 02:11:53.557632 27404 ts_tablet_manager.cc:1367] T 1162f9075da248d0a919db2eb45aa949 P cff7b7a19fc64db5849391b1d8e10cce: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.007s
I20240503 02:11:53.558640 27404 raft_consensus.cc:348] T 1162f9075da248d0a919db2eb45aa949 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.559991 27404 raft_consensus.cc:374] T 1162f9075da248d0a919db2eb45aa949 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.560245 27404 raft_consensus.cc:729] T 1162f9075da248d0a919db2eb45aa949 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.560708 27404 consensus_queue.cc:260] T 1162f9075da248d0a919db2eb45aa949 P cff7b7a19fc64db5849391b1d8e10cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.562134 27404 ts_tablet_manager.cc:1398] T 1162f9075da248d0a919db2eb45aa949 P cff7b7a19fc64db5849391b1d8e10cce: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:11:53.563664 27410 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap starting.
I20240503 02:11:53.566051 27410 tablet_bootstrap.cc:654] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.567170 27404 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce: Bootstrap starting.
I20240503 02:11:53.568373 27406 tablet_bootstrap.cc:492] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e: No bootstrap required, opened a new log
I20240503 02:11:53.568594 27406 ts_tablet_manager.cc:1367] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.015s
I20240503 02:11:53.569572 27406 raft_consensus.cc:348] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.569932 27406 raft_consensus.cc:374] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.570081 27406 raft_consensus.cc:729] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2835e59d40a48aea53c5dedb3bd1e9e, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.570438 27406 consensus_queue.cc:260] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.570830 27410 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f: No bootstrap required, opened a new log
I20240503 02:11:53.570448 27411 tablet_bootstrap.cc:492] T 1162f9075da248d0a919db2eb45aa949 P ae8b30bcfa5b425782a95374787dd230: Bootstrap starting.
I20240503 02:11:53.571111 27410 ts_tablet_manager.cc:1367] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 02:11:53.571394 27404 tablet_bootstrap.cc:654] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.571897 27410 raft_consensus.cc:348] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.572121 27410 raft_consensus.cc:374] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.572227 27410 raft_consensus.cc:729] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.572472 27410 consensus_queue.cc:260] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.573807 27410 ts_tablet_manager.cc:1398] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:53.574113 27410 tablet_bootstrap.cc:492] T 1162f9075da248d0a919db2eb45aa949 P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap starting.
I20240503 02:11:53.570191 27403 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb: Bootstrap starting.
I20240503 02:11:53.574812 27407 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221: No bootstrap required, opened a new log
I20240503 02:11:53.575371 27407 ts_tablet_manager.cc:1367] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.009s
I20240503 02:11:53.575712 27406 ts_tablet_manager.cc:1398] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:11:53.577314 27403 tablet_bootstrap.cc:654] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.577360 27411 tablet_bootstrap.cc:654] T 1162f9075da248d0a919db2eb45aa949 P ae8b30bcfa5b425782a95374787dd230: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.578764 27410 tablet_bootstrap.cc:654] T 1162f9075da248d0a919db2eb45aa949 P 2cf1cac003b44643a2d4de1204aacf0f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.577517 27407 raft_consensus.cc:348] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:53.578966 27407 raft_consensus.cc:374] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.579505 27407 raft_consensus.cc:729] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.579844 27407 consensus_queue.cc:260] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:53.580996 27407 ts_tablet_manager.cc:1398] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:11:53.581343 27407 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap starting.
I20240503 02:11:53.583343 27407 tablet_bootstrap.cc:654] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.586686 27406 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e: Bootstrap starting.
I20240503 02:11:53.587720 27403 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb: No bootstrap required, opened a new log
I20240503 02:11:53.587911 27403 ts_tablet_manager.cc:1367] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.004s
I20240503 02:11:53.588739 27407 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221: No bootstrap required, opened a new log
I20240503 02:11:53.588742 27406 tablet_bootstrap.cc:654] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.588681 27403 raft_consensus.cc:348] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.588948 27403 raft_consensus.cc:374] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.589365 27407 ts_tablet_manager.cc:1367] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20240503 02:11:53.589570 27403 raft_consensus.cc:729] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.589874 27403 consensus_queue.cc:260] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.591138 27410 tablet_bootstrap.cc:492] T 1162f9075da248d0a919db2eb45aa949 P 2cf1cac003b44643a2d4de1204aacf0f: No bootstrap required, opened a new log
I20240503 02:11:53.591302 27410 ts_tablet_manager.cc:1367] T 1162f9075da248d0a919db2eb45aa949 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240503 02:11:53.591451 27407 raft_consensus.cc:348] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.592078 27410 raft_consensus.cc:348] T 1162f9075da248d0a919db2eb45aa949 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.592314 27410 raft_consensus.cc:374] T 1162f9075da248d0a919db2eb45aa949 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.592384 27410 raft_consensus.cc:729] T 1162f9075da248d0a919db2eb45aa949 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.593039 27410 consensus_queue.cc:260] T 1162f9075da248d0a919db2eb45aa949 P 2cf1cac003b44643a2d4de1204aacf0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.594084 27411 tablet_bootstrap.cc:492] T 1162f9075da248d0a919db2eb45aa949 P ae8b30bcfa5b425782a95374787dd230: No bootstrap required, opened a new log
I20240503 02:11:53.594167 27410 ts_tablet_manager.cc:1398] T 1162f9075da248d0a919db2eb45aa949 P 2cf1cac003b44643a2d4de1204aacf0f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:53.594314 27411 ts_tablet_manager.cc:1367] T 1162f9075da248d0a919db2eb45aa949 P ae8b30bcfa5b425782a95374787dd230: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.003s
I20240503 02:11:53.594430 27410 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap starting.
I20240503 02:11:53.596765 27411 raft_consensus.cc:348] T 1162f9075da248d0a919db2eb45aa949 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.597080 27411 raft_consensus.cc:374] T 1162f9075da248d0a919db2eb45aa949 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.597222 27411 raft_consensus.cc:729] T 1162f9075da248d0a919db2eb45aa949 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.597558 27411 consensus_queue.cc:260] T 1162f9075da248d0a919db2eb45aa949 P ae8b30bcfa5b425782a95374787dd230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.598414 27411 ts_tablet_manager.cc:1398] T 1162f9075da248d0a919db2eb45aa949 P ae8b30bcfa5b425782a95374787dd230: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:11:53.598768 27411 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230: Bootstrap starting.
I20240503 02:11:53.598919 27403 ts_tablet_manager.cc:1398] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 02:11:53.599182 27403 tablet_bootstrap.cc:492] T 1162f9075da248d0a919db2eb45aa949 P 7c384005cb3041a0bd70e163473b6acb: Bootstrap starting.
I20240503 02:11:53.601557 27410 tablet_bootstrap.cc:654] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.603289 27404 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce: No bootstrap required, opened a new log
I20240503 02:11:53.603531 27404 ts_tablet_manager.cc:1367] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce: Time spent bootstrapping tablet: real 0.036s user 0.016s sys 0.016s
I20240503 02:11:53.605346 27403 tablet_bootstrap.cc:654] T 1162f9075da248d0a919db2eb45aa949 P 7c384005cb3041a0bd70e163473b6acb: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.605489 27398 raft_consensus.cc:482] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:53.605701 27398 raft_consensus.cc:504] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.606159 27404 raft_consensus.cc:348] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.606436 27404 raft_consensus.cc:374] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.606518 27404 raft_consensus.cc:729] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.606742 27404 consensus_queue.cc:260] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.607005 27398 leader_election.cc:288] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:11:53.607249 27404 ts_tablet_manager.cc:1398] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:11:53.608139 27403 tablet_bootstrap.cc:492] T 1162f9075da248d0a919db2eb45aa949 P 7c384005cb3041a0bd70e163473b6acb: No bootstrap required, opened a new log
I20240503 02:11:53.608283 27403 ts_tablet_manager.cc:1367] T 1162f9075da248d0a919db2eb45aa949 P 7c384005cb3041a0bd70e163473b6acb: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240503 02:11:53.608266 26397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1162f9075da248d0a919db2eb45aa949" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:11:53.608304 26657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1162f9075da248d0a919db2eb45aa949" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:11:53.608505 26397 raft_consensus.cc:2454] T 1162f9075da248d0a919db2eb45aa949 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 0.
I20240503 02:11:53.608552 26657 raft_consensus.cc:2454] T 1162f9075da248d0a919db2eb45aa949 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 0.
I20240503 02:11:53.609020 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1162f9075da248d0a919db2eb45aa949" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:11:53.608984 27403 raft_consensus.cc:348] T 1162f9075da248d0a919db2eb45aa949 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.609218 27403 raft_consensus.cc:374] T 1162f9075da248d0a919db2eb45aa949 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.609244 26138 raft_consensus.cc:2454] T 1162f9075da248d0a919db2eb45aa949 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 0.
I20240503 02:11:53.609292 27403 raft_consensus.cc:729] T 1162f9075da248d0a919db2eb45aa949 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.609622 26269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1162f9075da248d0a919db2eb45aa949" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:11:53.609520 27403 consensus_queue.cc:260] T 1162f9075da248d0a919db2eb45aa949 P 7c384005cb3041a0bd70e163473b6acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.609795 26269 raft_consensus.cc:2454] T 1162f9075da248d0a919db2eb45aa949 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 0.
I20240503 02:11:53.610653 26465 leader_election.cc:302] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 7c384005cb3041a0bd70e163473b6acb, ae8b30bcfa5b425782a95374787dd230, b2835e59d40a48aea53c5dedb3bd1e9e; no voters:
I20240503 02:11:53.610891 27398 raft_consensus.cc:2781] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:53.611014 27398 raft_consensus.cc:482] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:53.611146 27398 raft_consensus.cc:3037] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.611946 27406 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e: No bootstrap required, opened a new log
I20240503 02:11:53.612103 27406 ts_tablet_manager.cc:1367] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.002s
I20240503 02:11:53.613080 27403 ts_tablet_manager.cc:1398] T 1162f9075da248d0a919db2eb45aa949 P 7c384005cb3041a0bd70e163473b6acb: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:11:53.613338 27403 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb: Bootstrap starting.
I20240503 02:11:53.614878 27403 tablet_bootstrap.cc:654] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.616796 27410 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f: No bootstrap required, opened a new log
I20240503 02:11:53.616928 27410 ts_tablet_manager.cc:1367] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.004s
I20240503 02:11:53.617626 27410 raft_consensus.cc:348] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:53.617885 27410 raft_consensus.cc:374] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.617966 27410 raft_consensus.cc:729] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.618440 27410 consensus_queue.cc:260] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:53.620007 27410 ts_tablet_manager.cc:1398] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:53.621140 27398 raft_consensus.cc:504] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.622565 26397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1162f9075da248d0a919db2eb45aa949" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce"
I20240503 02:11:53.622838 26397 raft_consensus.cc:3037] T 1162f9075da248d0a919db2eb45aa949 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.624048 26986 raft_consensus.cc:482] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:53.624223 26986 raft_consensus.cc:504] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.625067 26986 leader_election.cc:288] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:11:53.625456 26269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d972dd11fcd9484f9a59f63d89f1e6e2" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:11:53.625540 26657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d972dd11fcd9484f9a59f63d89f1e6e2" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:11:53.625698 26269 raft_consensus.cc:2454] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 0.
W20240503 02:11:53.626071 26335 leader_election.cc:341] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:53.626124 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d972dd11fcd9484f9a59f63d89f1e6e2" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:11:53.626302 26138 raft_consensus.cc:2454] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 0.
I20240503 02:11:53.626700 26332 leader_election.cc:302] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, 7c384005cb3041a0bd70e163473b6acb, cff7b7a19fc64db5849391b1d8e10cce; no voters: ae8b30bcfa5b425782a95374787dd230
I20240503 02:11:53.625540 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d972dd11fcd9484f9a59f63d89f1e6e2" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:11:53.627107 26986 raft_consensus.cc:2781] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:53.627187 26986 raft_consensus.cc:482] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:53.627305 26986 raft_consensus.cc:3037] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.627911 27407 raft_consensus.cc:374] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.628054 27407 raft_consensus.cc:729] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.628509 27407 consensus_queue.cc:260] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.629019 26397 raft_consensus.cc:2454] T 1162f9075da248d0a919db2eb45aa949 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 1.
I20240503 02:11:53.629835 26986 raft_consensus.cc:504] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.630177 26008 raft_consensus.cc:2454] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 0.
I20240503 02:11:53.630357 27406 raft_consensus.cc:348] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:53.630688 27406 raft_consensus.cc:374] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.630769 27406 raft_consensus.cc:729] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2835e59d40a48aea53c5dedb3bd1e9e, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.631515 27407 ts_tablet_manager.cc:1398] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221: Time spent starting tablet: real 0.042s user 0.001s sys 0.001s
I20240503 02:11:53.631580 26657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1162f9075da248d0a919db2eb45aa949" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
I20240503 02:11:53.631829 26657 raft_consensus.cc:3037] T 1162f9075da248d0a919db2eb45aa949 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.632108 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d972dd11fcd9484f9a59f63d89f1e6e2" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
I20240503 02:11:53.632315 27411 tablet_bootstrap.cc:654] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230: Neither blocks nor log segments found. Creating new log.
W20240503 02:11:53.633136 26335 leader_election.cc:341] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:53.635268 26269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d972dd11fcd9484f9a59f63d89f1e6e2" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb"
I20240503 02:11:53.635524 26269 raft_consensus.cc:3037] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.635702 26657 raft_consensus.cc:2454] T 1162f9075da248d0a919db2eb45aa949 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 1.
I20240503 02:11:53.637951 26465 leader_election.cc:302] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: ae8b30bcfa5b425782a95374787dd230, b2835e59d40a48aea53c5dedb3bd1e9e, cff7b7a19fc64db5849391b1d8e10cce; no voters:
I20240503 02:11:53.638247 27130 raft_consensus.cc:2781] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:53.639190 26269 raft_consensus.cc:2454] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 1.
I20240503 02:11:53.640698 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d972dd11fcd9484f9a59f63d89f1e6e2" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221"
I20240503 02:11:53.640962 26008 raft_consensus.cc:3037] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.644516 26008 raft_consensus.cc:2454] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 1.
I20240503 02:11:53.645079 26332 leader_election.cc:302] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb, cff7b7a19fc64db5849391b1d8e10cce; no voters: ae8b30bcfa5b425782a95374787dd230
I20240503 02:11:53.645682 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1162f9075da248d0a919db2eb45aa949" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb"
I20240503 02:11:53.645874 26268 raft_consensus.cc:3037] T 1162f9075da248d0a919db2eb45aa949 P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.646375 27398 leader_election.cc:288] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [CANDIDATE]: Term 1 election: Requested vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:11:53.646389 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1162f9075da248d0a919db2eb45aa949" candidate_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f"
I20240503 02:11:53.646631 26138 raft_consensus.cc:3037] T 1162f9075da248d0a919db2eb45aa949 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.647580 27406 consensus_queue.cc:260] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:53.648159 27406 ts_tablet_manager.cc:1398] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240503 02:11:53.648749 26268 raft_consensus.cc:2454] T 1162f9075da248d0a919db2eb45aa949 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 1.
I20240503 02:11:53.650703 26138 raft_consensus.cc:2454] T 1162f9075da248d0a919db2eb45aa949 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2835e59d40a48aea53c5dedb3bd1e9e in term 1.
I20240503 02:11:53.652287 26986 leader_election.cc:288] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 1 election: Requested vote from peers ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:11:53.652655 27130 raft_consensus.cc:686] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 LEADER]: Becoming Leader. State: Replica: b2835e59d40a48aea53c5dedb3bd1e9e, State: Running, Role: LEADER
I20240503 02:11:53.652997 26986 raft_consensus.cc:2781] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:53.653093 27130 consensus_queue.cc:237] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.653573 26986 raft_consensus.cc:686] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 1 LEADER]: Becoming Leader. State: Replica: cff7b7a19fc64db5849391b1d8e10cce, State: Running, Role: LEADER
I20240503 02:11:53.654469 26986 consensus_queue.cc:237] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.657913 27411 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230: No bootstrap required, opened a new log
I20240503 02:11:53.658089 27411 ts_tablet_manager.cc:1367] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230: Time spent bootstrapping tablet: real 0.059s user 0.016s sys 0.009s
I20240503 02:11:53.658936 27403 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb: No bootstrap required, opened a new log
I20240503 02:11:53.659085 27403 ts_tablet_manager.cc:1367] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.022s
I20240503 02:11:53.659020 27411 raft_consensus.cc:348] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:53.659318 27411 raft_consensus.cc:374] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.659405 27411 raft_consensus.cc:729] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.659134 25859 catalog_manager.cc:5484] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e reported cstate change: term changed from 0 to 1, leader changed from <none> to b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197). New cstate: current_term: 1 leader_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:53.659672 27411 consensus_queue.cc:260] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:53.659778 27403 raft_consensus.cc:348] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:53.660013 27403 raft_consensus.cc:374] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.660107 27403 raft_consensus.cc:729] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.660372 27411 ts_tablet_manager.cc:1398] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:11:53.660341 27403 consensus_queue.cc:260] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:53.660663 27411 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230: Bootstrap starting.
I20240503 02:11:53.660992 27403 ts_tablet_manager.cc:1398] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:53.662887 27411 tablet_bootstrap.cc:654] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.665215 25858 catalog_manager.cc:5484] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: term changed from 0 to 1, leader changed from <none> to cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196). New cstate: current_term: 1 leader_uuid: "cff7b7a19fc64db5849391b1d8e10cce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:53.672921 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d972dd11fcd9484f9a59f63d89f1e6e2" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f"
I20240503 02:11:53.673166 26138 raft_consensus.cc:3037] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.679694 26138 raft_consensus.cc:2454] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 1.
I20240503 02:11:53.680769 27411 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230: No bootstrap required, opened a new log
I20240503 02:11:53.680907 27411 ts_tablet_manager.cc:1367] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.008s
I20240503 02:11:53.681408 27411 raft_consensus.cc:348] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.681607 27411 raft_consensus.cc:374] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.681699 27411 raft_consensus.cc:729] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.681900 27411 consensus_queue.cc:260] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.682384 27411 ts_tablet_manager.cc:1398] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:11:53.804898 27231 raft_consensus.cc:482] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:53.805099 27231 raft_consensus.cc:504] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:53.805969 27231 leader_election.cc:288] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369)
I20240503 02:11:53.806118 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e7ddb56be1499b94920ebd252595ce" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:11:53.806227 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e7ddb56be1499b94920ebd252595ce" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:11:53.806392 26138 raft_consensus.cc:2454] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 0.
I20240503 02:11:53.806440 26008 raft_consensus.cc:2454] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 0.
I20240503 02:11:53.806481 26529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e7ddb56be1499b94920ebd252595ce" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:11:53.806624 26657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e7ddb56be1499b94920ebd252595ce" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:11:53.806689 26529 raft_consensus.cc:2454] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 0.
I20240503 02:11:53.806828 26657 raft_consensus.cc:2454] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 0.
I20240503 02:11:53.806818 26202 leader_election.cc:302] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb; no voters:
I20240503 02:11:53.807170 27231 raft_consensus.cc:2781] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:53.807283 27231 raft_consensus.cc:482] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:53.807384 27231 raft_consensus.cc:3037] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.809149 27231 raft_consensus.cc:504] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:53.809876 27231 leader_election.cc:288] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 election: Requested vote from peers 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369)
I20240503 02:11:53.810094 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e7ddb56be1499b94920ebd252595ce" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f"
I20240503 02:11:53.810098 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e7ddb56be1499b94920ebd252595ce" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221"
I20240503 02:11:53.810282 26008 raft_consensus.cc:3037] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.810312 26138 raft_consensus.cc:3037] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.810362 26657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e7ddb56be1499b94920ebd252595ce" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
I20240503 02:11:53.810420 26529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e7ddb56be1499b94920ebd252595ce" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e"
I20240503 02:11:53.810525 26657 raft_consensus.cc:3037] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.810642 26529 raft_consensus.cc:3037] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.812978 26008 raft_consensus.cc:2454] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 1.
I20240503 02:11:53.813131 26138 raft_consensus.cc:2454] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 1.
I20240503 02:11:53.813196 26657 raft_consensus.cc:2454] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 1.
I20240503 02:11:53.813683 26202 leader_election.cc:302] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb; no voters:
I20240503 02:11:53.813539 26529 raft_consensus.cc:2454] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 1.
I20240503 02:11:53.814004 27231 raft_consensus.cc:2781] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:53.814759 27231 raft_consensus.cc:686] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Becoming Leader. State: Replica: 7c384005cb3041a0bd70e163473b6acb, State: Running, Role: LEADER
I20240503 02:11:53.815023 27231 consensus_queue.cc:237] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:53.817821 25858 catalog_manager.cc:5484] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c384005cb3041a0bd70e163473b6acb (127.23.104.195). New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:53.970120 27246 raft_consensus.cc:482] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:53.970286 27246 raft_consensus.cc:504] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:11:53.971117 27246 leader_election.cc:288] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:11:53.971465 26008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d972dd11fcd9484f9a59f63d89f1e6e2" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:11:53.971511 26397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d972dd11fcd9484f9a59f63d89f1e6e2" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:11:53.971714 26008 raft_consensus.cc:2379] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae8b30bcfa5b425782a95374787dd230 in current term 1: Already voted for candidate cff7b7a19fc64db5849391b1d8e10cce in this term.
I20240503 02:11:53.971825 26138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d972dd11fcd9484f9a59f63d89f1e6e2" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:11:53.972064 26138 raft_consensus.cc:2379] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae8b30bcfa5b425782a95374787dd230 in current term 1: Already voted for candidate cff7b7a19fc64db5849391b1d8e10cce in this term.
I20240503 02:11:53.972544 26592 leader_election.cc:302] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ae8b30bcfa5b425782a95374787dd230; no voters: 2cf1cac003b44643a2d4de1204aacf0f, 3bb103b2fd2344e7b55caf7389acf221, cff7b7a19fc64db5849391b1d8e10cce
I20240503 02:11:53.972518 26269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d972dd11fcd9484f9a59f63d89f1e6e2" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:11:53.972806 27246 raft_consensus.cc:3037] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.972860 26269 raft_consensus.cc:2379] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae8b30bcfa5b425782a95374787dd230 in current term 1: Already voted for candidate cff7b7a19fc64db5849391b1d8e10cce in this term.
I20240503 02:11:53.974990 27246 raft_consensus.cc:2726] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:11:54.039364 27130 consensus_queue.cc:1035] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:54.117290 27130 consensus_queue.cc:1035] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:54.121599 27130 consensus_queue.cc:1035] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:54.137640 27423 consensus_queue.cc:1035] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:54.166579 27424 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:54.172578 27424 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:54.199885 27430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:54.200299 27430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:11:54.200217 27442 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:54.200560 27442 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:54.215057 27430 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 02:11:54.244341 27231 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:54.275278 27231 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:54.291718 27428 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:54.294420 27428 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:54.330145 25941 consensus_peers.cc:485] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.199:34279: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
INSERT report
rows total: 2000
time total: 396.185 ms
time per row: 0.198092 ms
W20240503 02:11:54.744068 25941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.104.199:34279: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:11:54.761760 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.199:34279
--local_ip_for_outbound_sockets=127.23.104.199
--tserver_master_addrs=127.23.104.254:40709
--webserver_port=33273
--webserver_interface=127.23.104.199
--builtin_ntp_servers=127.23.104.212:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:11:55.084554 27509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:55.084839 27509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:55.084934 27509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:55.092816 27509 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:55.092923 27509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:55.093039 27509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.199
I20240503 02:11:55.102458 27509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.199:34279
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.23.104.199
--webserver_port=33273
--tserver_master_addrs=127.23.104.254:40709
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.199
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:55.103520 27509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:55.105127 27509 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:55.117277 27514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:55.117964 27515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:55.118654 27517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:55.119421 27509 server_base.cc:1034] running on GCE node
I20240503 02:11:55.119990 27509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:55.121022 27509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:55.122220 27509 hybrid_clock.cc:648] HybridClock initialized: now 1714702315122165 us; error 50 us; skew 500 ppm
I20240503 02:11:55.122506 27509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:55.124611 27509 webserver.cc:460] Webserver started at http://127.23.104.199:33273/ using document root <none> and password file <none>
I20240503 02:11:55.125339 27509 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:55.125479 27509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:55.130710 27509 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20240503 02:11:55.133347 27523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:55.134534 27509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 02:11:55.134675 27509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "0a801f0897884f108cdb89434b18628a"
format_stamp: "Formatted at 2024-05-03 02:11:43 on dist-test-slave-38ht"
I20240503 02:11:55.135442 27509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:55.155544 27509 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:55.156489 27509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:55.156795 27509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:55.157824 27509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:55.160032 27530 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 02:11:55.164955 27509 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 02:11:55.165040 27509 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20240503 02:11:55.165136 27509 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 02:11:55.167701 27509 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 02:11:55.167820 27509 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20240503 02:11:55.168143 27530 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a: Bootstrap starting.
I20240503 02:11:55.184888 27530 log.cc:824] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:55.187681 27530 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:55.188329 27530 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a: Bootstrap complete.
I20240503 02:11:55.189709 27530 ts_tablet_manager.cc:1367] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.011s
I20240503 02:11:55.200141 27530 raft_consensus.cc:348] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:55.201701 27530 raft_consensus.cc:729] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a801f0897884f108cdb89434b18628a, State: Initialized, Role: FOLLOWER
I20240503 02:11:55.202574 27530 consensus_queue.cc:260] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } }
I20240503 02:11:55.204589 27530 ts_tablet_manager.cc:1398] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
I20240503 02:11:55.219377 27509 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.199:34279
I20240503 02:11:55.219447 27638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.199:34279 every 8 connection(s)
I20240503 02:11:55.220906 27509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb
W20240503 02:11:55.222122 27640 tablet.cc:2369] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:55.228964 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 27509
I20240503 02:11:55.229264 23971 test_util.cc:268] Using random seed: 1046810683
I20240503 02:11:55.232893 27639 heartbeater.cc:347] Connected to a master server at 127.23.104.254:40709
I20240503 02:11:55.233182 27639 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:55.233929 27639 heartbeater.cc:511] Master 127.23.104.254:40709 requested a full tablet report, sending...
I20240503 02:11:55.236855 25858 ts_manager.cc:194] Re-registered known tserver with Master: 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279)
I20240503 02:11:55.238425 25858 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.199:60681
I20240503 02:11:55.245606 23971 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 02:11:55.245985 23971 test_util.cc:268] Using random seed: 1046827402
I20240503 02:11:55.259487 23971 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 02:11:55.260071 23971 test_util.cc:268] Using random seed: 1046841478
I20240503 02:11:55.301934 23971 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 02:11:55.312891 23971 test_util.cc:268] Using random seed: 1046894298
I20240503 02:11:55.412714 27639 heartbeater.cc:503] Master 127.23.104.254:40709 was elected leader, sending a full tablet report...
I20240503 02:11:55.462206 23971 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 02:11:55.526191 23971 test_util.cc:268] Using random seed: 1047107601
I20240503 02:11:55.694218 23971 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240503 02:11:55.725016 26594 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 108530us
W20240503 02:11:55.979383 26465 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 53020.4us
W20240503 02:11:57.668701 26335 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 120898us
W20240503 02:11:57.974835 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 45267.3us
W20240503 02:11:58.340657 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 147164us
W20240503 02:11:59.155874 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 403015us
W20240503 02:11:59.292579 27766 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.055s sys 0.079s
W20240503 02:11:59.292796 27766 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.580s user 0.055s sys 0.079s
W20240503 02:11:59.730183 26396 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.030s sys 0.049s
W20240503 02:11:59.730444 26396 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.585s user 0.030s sys 0.049s
W20240503 02:11:59.747570 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 188550us
W20240503 02:12:00.183261 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 46200.8us
I20240503 02:12:00.213835 27573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:00.272106 26378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:00.326275 26247 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:00.342681 25988 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.424263 26117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 02:12:00.469115 26595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 85048.8us
I20240503 02:12:00.636601 26509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.753584 26638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 02:12:01.233083 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 66714.1us
W20240503 02:12:01.360193 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 44281.7us
W20240503 02:12:02.801715 26595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 78729.6us
W20240503 02:12:02.821038 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 139848us
W20240503 02:12:02.840637 26594 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 103010us
W20240503 02:12:03.238853 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 103324us
W20240503 02:12:03.730619 25944 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 236980us
I20240503 02:12:04.275115 27573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:12:04.366499 26594 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 52031.4us
W20240503 02:12:04.386879 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 81409us
I20240503 02:12:04.454784 26117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:04.872400 26247 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:04.925865 26638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.257720 26509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.319681 25988 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.330332 26378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:12:05.680248 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 53184.3us
W20240503 02:12:06.018780 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 48086.4us
W20240503 02:12:06.309732 26074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 50128.9us
I20240503 02:12:06.437110 26395 consensus_queue.cc:237] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 1.77, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
W20240503 02:12:06.442492 26335 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 253325us
I20240503 02:12:06.445397 26653 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240503 02:12:06.455682 28031 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20240503 02:12:06.476322 26136 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240503 02:12:06.476809 26004 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240503 02:12:06.478576 26268 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240503 02:12:06.497025 28119 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20240503 02:12:06.521394 26267 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.78: config changed from index -1 to 78, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:06.540441 28119 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.005s
I20240503 02:12:06.559329 25861 catalog_manager.cc:5484] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb reported cstate change: config changed from index -1 to 78, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "cff7b7a19fc64db5849391b1d8e10cce" committed_config { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:06.575238 28030 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.035s
I20240503 02:12:06.582571 26005 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.78: config changed from index -1 to 78, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:06.600051 26135 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.78: config changed from index -1 to 78, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:06.542335 27970 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 1 LEADER]: Committing config change with OpId 1.78: config changed from index -1 to 78, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
W20240503 02:12:06.629963 26332 consensus_peers.cc:485] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d972dd11fcd9484f9a59f63d89f1e6e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:06.692153 26658 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.78: config changed from index -1 to 78, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
W20240503 02:12:06.738051 26334 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 144422us
W20240503 02:12:07.161083 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 475588us
I20240503 02:12:07.210459 28155 ts_tablet_manager.cc:902] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a: Initiating tablet copy from peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675)
I20240503 02:12:07.245482 28155 tablet_copy_client.cc:308] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:43675
I20240503 02:12:07.366171 26003 consensus_queue.cc:237] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: true } }
I20240503 02:12:07.370024 26264 raft_consensus.cc:1261] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 02:12:07.370632 28094 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 02:12:07.373971 26528 raft_consensus.cc:1261] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 02:12:07.375450 28094 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 02:12:07.377331 26419 tablet_copy_service.cc:140] P cff7b7a19fc64db5849391b1d8e10cce: Received BeginTabletCopySession request for tablet d972dd11fcd9484f9a59f63d89f1e6e2 from peer 0a801f0897884f108cdb89434b18628a ({username='slave'} at 127.23.104.199:37613)
I20240503 02:12:07.377609 26419 tablet_copy_service.cc:161] P cff7b7a19fc64db5849391b1d8e10cce: Beginning new tablet copy session on tablet d972dd11fcd9484f9a59f63d89f1e6e2 from peer 0a801f0897884f108cdb89434b18628a at {username='slave'} at 127.23.104.199:37613: session id = 0a801f0897884f108cdb89434b18628a-d972dd11fcd9484f9a59f63d89f1e6e2
I20240503 02:12:07.379532 28112 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: true } } }
I20240503 02:12:07.382200 26419 tablet_copy_source_session.cc:205] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:07.384541 26264 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: true } } }
I20240503 02:12:07.385645 28155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d972dd11fcd9484f9a59f63d89f1e6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:07.388984 26658 raft_consensus.cc:1261] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 02:12:07.388748 25861 catalog_manager.cc:5484] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 reported cstate change: config changed from index -1 to 84, NON_VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) added. New cstate: current_term: 1 leader_uuid: "3bb103b2fd2344e7b55caf7389acf221" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:07.390166 26392 raft_consensus.cc:1261] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 02:12:07.401314 28155 tablet_copy_client.cc:783] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:07.408533 26527 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: true } } }
I20240503 02:12:07.413833 28155 tablet_copy_client.cc:647] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:07.415604 28112 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 02:12:07.416586 28155 tablet_copy_client.cc:523] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:07.419970 28112 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 02:12:07.426273 26658 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: true } } }
I20240503 02:12:07.424394 28155 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a: Bootstrap starting.
I20240503 02:12:07.430831 26391 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: true } } }
W20240503 02:12:07.537652 25941 consensus_peers.cc:485] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 -> Peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): Couldn't send request to peer 2cf1cac003b44643a2d4de1204aacf0f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e866b4f1b338419ab1025287b2aaac5e. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:07.813856 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 333271us
W20240503 02:12:07.814080 26465 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 91334.9us
W20240503 02:12:08.140007 26594 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 74467.3us
W20240503 02:12:08.169749 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 310867us
W20240503 02:12:08.184968 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 308810us
W20240503 02:12:08.223964 26594 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 46320.8us
W20240503 02:12:08.271660 27959 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.013s sys 0.097s
W20240503 02:12:08.271904 27959 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.568s user 0.013s sys 0.098s
W20240503 02:12:08.278851 28123 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.004s sys 0.008s
W20240503 02:12:08.278967 28123 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.576s user 0.004s sys 0.008s
I20240503 02:12:08.320660 28155 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a: Bootstrap replayed 1/1 log segments. Stats: ops{read=85 overwritten=0 applied=85 ignored=0} inserts{seen=770 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:08.321120 28155 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a: Bootstrap complete.
I20240503 02:12:08.321383 28155 ts_tablet_manager.cc:1367] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a: Time spent bootstrapping tablet: real 0.897s user 0.140s sys 0.012s
I20240503 02:12:08.322373 28155 raft_consensus.cc:348] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:08.322644 28155 raft_consensus.cc:729] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0a801f0897884f108cdb89434b18628a, State: Initialized, Role: LEARNER
I20240503 02:12:08.322821 28155 consensus_queue.cc:260] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 85, Last appended: 1.85, Last appended by leader: 85, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:08.328608 28155 ts_tablet_manager.cc:1398] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:12:08.331553 26419 tablet_copy_service.cc:342] P cff7b7a19fc64db5849391b1d8e10cce: Request end of tablet copy session 0a801f0897884f108cdb89434b18628a-d972dd11fcd9484f9a59f63d89f1e6e2 received from {username='slave'} at 127.23.104.199:37613
I20240503 02:12:08.331714 26419 tablet_copy_service.cc:433] P cff7b7a19fc64db5849391b1d8e10cce: ending tablet copy session 0a801f0897884f108cdb89434b18628a-d972dd11fcd9484f9a59f63d89f1e6e2 on tablet d972dd11fcd9484f9a59f63d89f1e6e2 with peer 0a801f0897884f108cdb89434b18628a
I20240503 02:12:08.379890 28203 ts_tablet_manager.cc:902] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f: Initiating tablet copy from peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143)
I20240503 02:12:08.574828 27594 raft_consensus.cc:1203] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.77->[1.78-1.96] Dedup: 1.85->[1.86-1.96]
I20240503 02:12:08.615499 28210 raft_consensus.cc:1050] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce: attempting to promote NON_VOTER 0a801f0897884f108cdb89434b18628a to VOTER
I20240503 02:12:08.616623 28210 consensus_queue.cc:237] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 1.96, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:08.621258 26005 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240503 02:12:08.628401 28118 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20240503 02:12:08.633754 27594 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240503 02:12:08.635412 28210 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20240503 02:12:08.641515 26137 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240503 02:12:08.650352 26659 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240503 02:12:08.655028 26269 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240503 02:12:08.658224 28210 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20240503 02:12:08.658471 28030 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 02:12:08.663463 26393 consensus_queue.cc:237] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:08.666622 28210 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20240503 02:12:08.671082 28030 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 1 LEADER]: Committing config change with OpId 1.97: config changed from index 78 to 97, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:08.676136 26332 consensus_peers.cc:485] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39d3ba2cbbad40d6a232e534c65d2d74. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:08.678498 26004 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 78 to 97, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:08.679553 26659 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 02:12:08.680300 26005 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 02:12:08.680948 28210 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 02:12:08.683374 26265 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 02:12:08.685024 25861 catalog_manager.cc:5484] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: config changed from index 78 to 97, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cff7b7a19fc64db5849391b1d8e10cce" committed_config { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:08.688318 28203 tablet_copy_client.cc:308] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.193:34143
I20240503 02:12:08.696204 26136 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240503 02:12:08.699250 28037 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 02:12:08.700224 26028 tablet_copy_service.cc:140] P 3bb103b2fd2344e7b55caf7389acf221: Received BeginTabletCopySession request for tablet e866b4f1b338419ab1025287b2aaac5e from peer 2cf1cac003b44643a2d4de1204aacf0f ({username='slave'} at 127.23.104.194:52309)
I20240503 02:12:08.700526 26028 tablet_copy_service.cc:161] P 3bb103b2fd2344e7b55caf7389acf221: Beginning new tablet copy session on tablet e866b4f1b338419ab1025287b2aaac5e from peer 2cf1cac003b44643a2d4de1204aacf0f at {username='slave'} at 127.23.104.194:52309: session id = 2cf1cac003b44643a2d4de1204aacf0f-e866b4f1b338419ab1025287b2aaac5e
I20240503 02:12:08.710827 26659 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 78 to 97, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:08.711445 28141 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 02:12:08.720647 28118 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 02:12:08.721179 26267 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 78 to 97, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:08.725286 26028 tablet_copy_source_session.cc:205] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:08.728519 27594 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 78 to 97, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:08.735910 28118 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:08.738210 26139 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 78 to 97, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:08.743856 26004 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:08.749661 28203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e866b4f1b338419ab1025287b2aaac5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:08.761078 28203 tablet_copy_client.cc:783] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:08.765233 26656 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:08.766693 26136 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:08.773000 28203 tablet_copy_client.cc:647] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:08.799687 28203 tablet_copy_client.cc:523] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:08.822257 28203 tablet_bootstrap.cc:492] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap starting.
I20240503 02:12:09.151064 25862 catalog_manager.cc:5484] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f reported cstate change: config changed from index -1 to 79, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "cff7b7a19fc64db5849391b1d8e10cce" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:09.216634 28248 ts_tablet_manager.cc:902] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a: Initiating tablet copy from peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675)
I20240503 02:12:09.241569 26397 consensus_queue.cc:237] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:09.248828 26001 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240503 02:12:09.249882 28030 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240503 02:12:09.254276 27594 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240503 02:12:09.255185 28210 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240503 02:12:09.256597 26659 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240503 02:12:09.257484 28030 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240503 02:12:09.257930 28248 tablet_copy_client.cc:308] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:43675
I20240503 02:12:09.259047 26419 tablet_copy_service.cc:140] P cff7b7a19fc64db5849391b1d8e10cce: Received BeginTabletCopySession request for tablet 39d3ba2cbbad40d6a232e534c65d2d74 from peer 0a801f0897884f108cdb89434b18628a ({username='slave'} at 127.23.104.199:37613)
I20240503 02:12:09.259341 26419 tablet_copy_service.cc:161] P cff7b7a19fc64db5849391b1d8e10cce: Beginning new tablet copy session on tablet 39d3ba2cbbad40d6a232e534c65d2d74 from peer 0a801f0897884f108cdb89434b18628a at {username='slave'} at 127.23.104.199:37613: session id = 0a801f0897884f108cdb89434b18628a-39d3ba2cbbad40d6a232e534c65d2d74
I20240503 02:12:09.265187 28037 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index 97 to 105, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:09.267282 26001 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 97 to 105, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:09.270190 25853 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d972dd11fcd9484f9a59f63d89f1e6e2 with cas_config_opid_index 97: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:09.272122 27594 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 97 to 105, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:09.273241 25863 catalog_manager.cc:5484] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: config changed from index 97 to 105, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "cff7b7a19fc64db5849391b1d8e10cce" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:09.278736 26659 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 97 to 105, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:09.315380 26419 tablet_copy_source_session.cc:205] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:09.328763 28248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d3ba2cbbad40d6a232e534c65d2d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:09.337924 28248 tablet_copy_client.cc:783] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:09.338234 28248 tablet_copy_client.cc:647] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 1 WAL segments...
W20240503 02:12:09.342413 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 131923us
I20240503 02:12:09.342554 26135 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240503 02:12:09.356099 28248 tablet_copy_client.cc:523] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:09.381500 26265 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:09.426440 28248 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a: Bootstrap starting.
I20240503 02:12:09.441439 28030 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 102, Time since last communication: 0.000s
I20240503 02:12:09.455046 26247 tablet_service.cc:1514] Processing DeleteTablet for tablet d972dd11fcd9484f9a59f63d89f1e6e2 with delete_type TABLET_DATA_TOMBSTONED (TS 7c384005cb3041a0bd70e163473b6acb not found in new config with opid_index 105) from {username='slave'} at 127.0.0.1:37992
W20240503 02:12:09.559159 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 87767us
I20240503 02:12:09.626094 28266 tablet_replica.cc:330] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb: stopping tablet replica
I20240503 02:12:09.626688 28266 raft_consensus.cc:2229] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:09.631834 28248 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a: Bootstrap replayed 1/1 log segments. Stats: ops{read=80 overwritten=0 applied=80 ignored=0} inserts{seen=765 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:09.632336 28248 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a: Bootstrap complete.
I20240503 02:12:09.632714 28248 ts_tablet_manager.cc:1367] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a: Time spent bootstrapping tablet: real 0.206s user 0.136s sys 0.021s
I20240503 02:12:09.633666 28248 raft_consensus.cc:348] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:09.634014 28248 raft_consensus.cc:729] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0a801f0897884f108cdb89434b18628a, State: Initialized, Role: LEARNER
I20240503 02:12:09.634289 28248 consensus_queue.cc:260] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 80, Last appended: 1.80, Last appended by leader: 80, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:09.634923 28248 ts_tablet_manager.cc:1398] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:09.635633 26419 tablet_copy_service.cc:342] P cff7b7a19fc64db5849391b1d8e10cce: Request end of tablet copy session 0a801f0897884f108cdb89434b18628a-39d3ba2cbbad40d6a232e534c65d2d74 received from {username='slave'} at 127.23.104.199:37613
I20240503 02:12:09.635766 26419 tablet_copy_service.cc:433] P cff7b7a19fc64db5849391b1d8e10cce: ending tablet copy session 0a801f0897884f108cdb89434b18628a-39d3ba2cbbad40d6a232e534c65d2d74 on tablet 39d3ba2cbbad40d6a232e534c65d2d74 with peer 0a801f0897884f108cdb89434b18628a
I20240503 02:12:09.639842 28266 pending_rounds.cc:62] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb: Trying to abort 1 pending ops.
I20240503 02:12:09.640019 28266 pending_rounds.cc:68] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb: Aborting op as it isn't in flight: id { term: 1 index: 100 } timestamp: 7023420739327500288 op_type: WRITE_OP write_request { tablet_id: "d972dd11fcd9484f9a59f63d89f1e6e2" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\036\243\314^i\231&\343\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""f8747d9b4fc4d023" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023420739233906688 authz_token { token_data: "\010\227\222\321\261\006\"3\n\005slave\022*\n 6e0193858acc4526bc2d4a6f4918b283\020\001\030\001 \001(\001" signature: "<redacted>""\232\021m\355\0174\305\357\376\364&^I\347\302+\261\014K\256\335\274\220\333C\002z\260\307;\206\253;kP}\330q\313\361\301Q4\330\222\005\364\036\003\362\234g%\265\225\r5I&\200P\327\347\215" signing_key_seq_num: 0 } } request_id { client_id: "aa5d1cf5815349fd826312d67e9703d7" seq_no: 286 first_incomplete_seq_no: 286 attempt_no: 0 }
I20240503 02:12:09.646389 28266 raft_consensus.cc:2258] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:09.649163 28266 ts_tablet_manager.cc:1875] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:09.661468 28266 ts_tablet_manager.cc:1888] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.100
I20240503 02:12:09.661780 28266 log.cc:1196] T d972dd11fcd9484f9a59f63d89f1e6e2 P 7c384005cb3041a0bd70e163473b6acb: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/d972dd11fcd9484f9a59f63d89f1e6e2
I20240503 02:12:09.667863 25853 catalog_manager.cc:4830] TS 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999): tablet d972dd11fcd9484f9a59f63d89f1e6e2 (table rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]) successfully deleted
I20240503 02:12:09.685662 27594 raft_consensus.cc:1203] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.78->[1.79-1.81] Dedup: 1.80->[1.81-1.81]
W20240503 02:12:09.817785 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 256772us
W20240503 02:12:09.847530 26074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 181245us
I20240503 02:12:09.863688 26135 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 97 to 105, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:10.006475 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 82214us
I20240503 02:12:10.028159 26137 consensus_queue.cc:237] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 146, Committed index: 146, Last appended: 1.146, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: true } }
W20240503 02:12:10.036731 26072 consensus_peers.cc:485] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f -> Peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Couldn't send request to peer 3bb103b2fd2344e7b55caf7389acf221. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 041ffbedf2c04dbea4661b4c33f78ac1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:10.065095 26269 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 146. Preceding OpId from leader: term: 1 index: 147. (index mismatch)
I20240503 02:12:10.066466 26397 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 146. Preceding OpId from leader: term: 1 index: 147. (index mismatch)
I20240503 02:12:10.106810 28200 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 147, Last known committed idx: 146, Time since last communication: 0.000s
I20240503 02:12:10.107342 28272 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 147, Last known committed idx: 146, Time since last communication: 0.000s
I20240503 02:12:10.145587 26656 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 146. Preceding OpId from leader: term: 1 index: 147. (index mismatch)
I20240503 02:12:10.151480 26527 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 146. Preceding OpId from leader: term: 1 index: 147. (index mismatch)
I20240503 02:12:10.160604 28272 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 147, Last known committed idx: 146, Time since last communication: 0.000s
I20240503 02:12:10.160970 27980 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 147, Last known committed idx: 146, Time since last communication: 0.000s
W20240503 02:12:10.203676 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 331102us
I20240503 02:12:10.209317 27332 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Committing config change with OpId 1.147: config changed from index -1 to 147, NON_VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) added. New config: { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: true } } }
W20240503 02:12:10.214699 26074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 70242.3us
I20240503 02:12:10.222484 26269 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.147: config changed from index -1 to 147, NON_VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) added. New config: { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: true } } }
I20240503 02:12:10.227103 25859 catalog_manager.cc:5484] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f reported cstate change: config changed from index -1 to 147, NON_VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) added. New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:10.239627 26527 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.147: config changed from index -1 to 147, NON_VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) added. New config: { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: true } } }
I20240503 02:12:10.248184 26392 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.147: config changed from index -1 to 147, NON_VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) added. New config: { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: true } } }
W20240503 02:12:10.251897 26334 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 86238.1us
I20240503 02:12:10.275476 26659 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.147: config changed from index -1 to 147, NON_VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) added. New config: { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: true } } }
W20240503 02:12:10.312242 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 156547us
W20240503 02:12:10.313031 26335 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 121390us
W20240503 02:12:10.325289 26074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 44073.5us
I20240503 02:12:10.348225 28292 raft_consensus.cc:1050] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce: attempting to promote NON_VOTER 0a801f0897884f108cdb89434b18628a to VOTER
I20240503 02:12:10.349309 28292 consensus_queue.cc:237] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:10.355525 26139 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 02:12:10.357280 27593 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 02:12:10.361524 26653 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 02:12:10.362993 28299 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 82, Time since last communication: 0.000s
I20240503 02:12:10.365254 28299 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 02:12:10.370666 28299 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 02:12:10.372457 26264 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 02:12:10.390251 28307 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index 79 to 84, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:10.396811 27593 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 79 to 84, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:10.396947 26139 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 79 to 84, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:10.401872 26653 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 79 to 84, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:10.404417 26005 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 02:12:10.407021 25866 catalog_manager.cc:5484] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: config changed from index 79 to 84, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cff7b7a19fc64db5849391b1d8e10cce" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:10.436882 28303 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 82, Time since last communication: 0.000s
W20240503 02:12:10.461671 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 58681.4us
I20240503 02:12:10.467195 28300 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
W20240503 02:12:10.476207 26334 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 45934.9us
I20240503 02:12:10.482225 26269 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 79 to 84, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:10.461813 26004 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 79 to 84, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:10.540364 25943 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 44306.5us
W20240503 02:12:10.579700 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 80588.6us
W20240503 02:12:10.588228 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 118541us
I20240503 02:12:10.634687 28323 ts_tablet_manager.cc:902] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221: Initiating tablet copy from peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:12:10.665771 28203 tablet_bootstrap.cc:492] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=87 overwritten=0 applied=87 ignored=0} inserts{seen=772 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:10.666311 28203 tablet_bootstrap.cc:492] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f: Bootstrap complete.
I20240503 02:12:10.666711 28203 ts_tablet_manager.cc:1367] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f: Time spent bootstrapping tablet: real 1.845s user 0.143s sys 0.005s
I20240503 02:12:10.667585 28203 raft_consensus.cc:348] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: true } }
I20240503 02:12:10.667886 28203 raft_consensus.cc:729] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2cf1cac003b44643a2d4de1204aacf0f, State: Initialized, Role: LEARNER
I20240503 02:12:10.668110 28203 consensus_queue.cc:260] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 87, Last appended: 1.87, Last appended by leader: 87, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: true } }
I20240503 02:12:10.668809 28203 ts_tablet_manager.cc:1398] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:12:10.670454 26028 tablet_copy_service.cc:342] P 3bb103b2fd2344e7b55caf7389acf221: Request end of tablet copy session 2cf1cac003b44643a2d4de1204aacf0f-e866b4f1b338419ab1025287b2aaac5e received from {username='slave'} at 127.23.104.194:52309
I20240503 02:12:10.670614 26028 tablet_copy_service.cc:433] P 3bb103b2fd2344e7b55caf7389acf221: ending tablet copy session 2cf1cac003b44643a2d4de1204aacf0f-e866b4f1b338419ab1025287b2aaac5e on tablet e866b4f1b338419ab1025287b2aaac5e with peer 2cf1cac003b44643a2d4de1204aacf0f
I20240503 02:12:10.689291 26392 consensus_queue.cc:237] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.86, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:10.705020 27593 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240503 02:12:10.705142 26137 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240503 02:12:10.705555 26267 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240503 02:12:10.705725 28303 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240503 02:12:10.706209 28303 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240503 02:12:10.706641 28303 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240503 02:12:10.712373 28292 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 1 LEADER]: Committing config change with OpId 1.87: config changed from index 84 to 87, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:10.712728 26137 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 84 to 87, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:10.716657 25853 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39d3ba2cbbad40d6a232e534c65d2d74 with cas_config_opid_index 84: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:10.716521 26267 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 84 to 87, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:10.719089 27593 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 84 to 87, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:10.720983 25866 catalog_manager.cc:5484] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: config changed from index 84 to 87, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New cstate: current_term: 1 leader_uuid: "cff7b7a19fc64db5849391b1d8e10cce" committed_config { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:10.727273 26638 tablet_service.cc:1514] Processing DeleteTablet for tablet 39d3ba2cbbad40d6a232e534c65d2d74 with delete_type TABLET_DATA_TOMBSTONED (TS ae8b30bcfa5b425782a95374787dd230 not found in new config with opid_index 87) from {username='slave'} at 127.0.0.1:56314
W20240503 02:12:10.735299 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 59281.7us
I20240503 02:12:10.769197 26004 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240503 02:12:10.783977 26137 raft_consensus.cc:1203] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.83->[1.84-1.96] Dedup: 1.87->[1.88-1.96]
I20240503 02:12:10.792775 28292 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240503 02:12:10.794735 26001 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 84 to 87, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:10.821023 28323 tablet_copy_client.cc:308] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:39753
I20240503 02:12:10.840333 26159 tablet_copy_service.cc:140] P 2cf1cac003b44643a2d4de1204aacf0f: Received BeginTabletCopySession request for tablet 041ffbedf2c04dbea4661b4c33f78ac1 from peer 3bb103b2fd2344e7b55caf7389acf221 ({username='slave'} at 127.23.104.193:35363)
I20240503 02:12:10.840581 26159 tablet_copy_service.cc:161] P 2cf1cac003b44643a2d4de1204aacf0f: Beginning new tablet copy session on tablet 041ffbedf2c04dbea4661b4c33f78ac1 from peer 3bb103b2fd2344e7b55caf7389acf221 at {username='slave'} at 127.23.104.193:35363: session id = 3bb103b2fd2344e7b55caf7389acf221-041ffbedf2c04dbea4661b4c33f78ac1
I20240503 02:12:10.845644 26159 tablet_copy_source_session.cc:205] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:10.851058 28323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041ffbedf2c04dbea4661b4c33f78ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:10.864877 28327 tablet_replica.cc:330] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230: stopping tablet replica
I20240503 02:12:10.921066 28327 raft_consensus.cc:2229] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:10.921415 28327 raft_consensus.cc:2258] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:10.934254 26267 consensus_queue.cc:237] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.102, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:10.944825 26397 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240503 02:12:10.945935 28225 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240503 02:12:10.948138 26528 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240503 02:12:10.948894 28236 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240503 02:12:10.952762 28236 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Committing config change with OpId 1.103: config changed from index -1 to 103, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:10.956423 26527 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index -1 to 103, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:10.961942 28327 ts_tablet_manager.cc:1875] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:12:10.971995 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 113715us
I20240503 02:12:10.986827 25859 catalog_manager.cc:5484] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e reported cstate change: config changed from index -1 to 103, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:10.991194 28286 raft_consensus.cc:1050] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221: attempting to promote NON_VOTER 2cf1cac003b44643a2d4de1204aacf0f to VOTER
I20240503 02:12:11.003345 28286 consensus_queue.cc:237] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 1.98, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } }
I20240503 02:12:11.006320 26139 raft_consensus.cc:1261] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEARNER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 02:12:11.006969 28244 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240503 02:12:11.007083 26391 raft_consensus.cc:1261] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 02:12:11.007552 28244 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240503 02:12:11.009523 26268 raft_consensus.cc:1261] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 02:12:11.011703 28242 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240503 02:12:11.013170 26524 raft_consensus.cc:1261] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 02:12:11.013170 26397 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index -1 to 103, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:11.016872 28242 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [term 1 LEADER]: Committing config change with OpId 1.99: config changed from index 84 to 99, 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } } }
I20240503 02:12:11.021610 26268 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 84 to 99, 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } } }
I20240503 02:12:11.022320 28286 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240503 02:12:11.024142 26525 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 84 to 99, 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } } }
I20240503 02:12:11.026101 26657 raft_consensus.cc:1261] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 02:12:11.033435 28241 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 96, Time since last communication: 0.000s
I20240503 02:12:11.040534 26137 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 84 to 99, 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } } }
I20240503 02:12:11.048537 25863 catalog_manager.cc:5484] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 reported cstate change: config changed from index 84 to 99, 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3bb103b2fd2344e7b55caf7389acf221" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:11.056905 26397 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 84 to 99, 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } } }
I20240503 02:12:11.035743 26653 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240503 02:12:11.074868 26005 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240503 02:12:11.081133 28236 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240503 02:12:11.090500 26653 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index -1 to 103, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:11.155730 28225 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240503 02:12:11.209558 28327 ts_tablet_manager.cc:1888] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.86
I20240503 02:12:11.211848 28323 tablet_copy_client.cc:783] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:11.212126 28327 log.cc:1196] T 39d3ba2cbbad40d6a232e534c65d2d74 P ae8b30bcfa5b425782a95374787dd230: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal/wals/39d3ba2cbbad40d6a232e534c65d2d74
I20240503 02:12:11.212522 28323 tablet_copy_client.cc:647] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:11.213635 25854 catalog_manager.cc:4830] TS ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369): tablet 39d3ba2cbbad40d6a232e534c65d2d74 (table rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]) successfully deleted
I20240503 02:12:11.222343 28323 tablet_copy_client.cc:523] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:11.239403 28323 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap starting.
I20240503 02:12:11.255695 26005 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index -1 to 103, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:11.367269 26656 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 84 to 99, 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } } }
W20240503 02:12:11.412739 26202 consensus_peers.cc:485] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c27502976d474b938cce2235e5784f79. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:11.524482 26007 consensus_queue.cc:237] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 1.102, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } }
I20240503 02:12:11.528055 26397 raft_consensus.cc:1261] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240503 02:12:11.529148 28242 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 101, Time since last communication: 0.000s
I20240503 02:12:11.535297 26656 raft_consensus.cc:1261] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240503 02:12:11.544328 28242 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 101, Time since last communication: 0.000s
I20240503 02:12:11.547140 26269 raft_consensus.cc:1261] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240503 02:12:11.570147 28242 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 101, Time since last communication: 0.000s
I20240503 02:12:11.577392 28357 ts_tablet_manager.cc:902] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a: Initiating tablet copy from peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999)
I20240503 02:12:11.611654 26133 raft_consensus.cc:1261] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240503 02:12:11.626015 28242 consensus_queue.cc:1035] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 101, Time since last communication: 0.000s
I20240503 02:12:11.633893 28244 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 [term 1 LEADER]: Committing config change with OpId 1.103: config changed from index 99 to 103, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } } }
I20240503 02:12:11.638449 25851 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e866b4f1b338419ab1025287b2aaac5e with cas_config_opid_index 99: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:11.645972 25859 catalog_manager.cc:5484] T e866b4f1b338419ab1025287b2aaac5e P 3bb103b2fd2344e7b55caf7389acf221 reported cstate change: config changed from index 99 to 103, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New cstate: current_term: 1 leader_uuid: "3bb103b2fd2344e7b55caf7389acf221" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:11.658550 26134 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 99 to 103, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } } }
I20240503 02:12:11.661620 26399 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 99 to 103, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } } }
I20240503 02:12:11.742659 26653 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 99 to 103, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } } }
W20240503 02:12:11.791574 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 50104.2us
W20240503 02:12:11.791947 26202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 44392.4us
I20240503 02:12:11.811115 28357 tablet_copy_client.cc:308] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:45999
I20240503 02:12:11.879549 26267 raft_consensus.cc:2932] T e866b4f1b338419ab1025287b2aaac5e P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 99 to 103, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } } }
I20240503 02:12:11.910799 26289 tablet_copy_service.cc:140] P 7c384005cb3041a0bd70e163473b6acb: Received BeginTabletCopySession request for tablet c27502976d474b938cce2235e5784f79 from peer 0a801f0897884f108cdb89434b18628a ({username='slave'} at 127.23.104.199:38513)
I20240503 02:12:11.911064 26289 tablet_copy_service.cc:161] P 7c384005cb3041a0bd70e163473b6acb: Beginning new tablet copy session on tablet c27502976d474b938cce2235e5784f79 from peer 0a801f0897884f108cdb89434b18628a at {username='slave'} at 127.23.104.199:38513: session id = 0a801f0897884f108cdb89434b18628a-c27502976d474b938cce2235e5784f79
I20240503 02:12:11.914988 26289 tablet_copy_source_session.cc:205] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:11.924245 28357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c27502976d474b938cce2235e5784f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:11.932340 28357 tablet_copy_client.cc:783] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:11.932569 28357 tablet_copy_client.cc:647] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:11.939579 28357 tablet_copy_client.cc:523] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:11.946939 28357 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a: Bootstrap starting.
W20240503 02:12:12.038235 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 81834us
I20240503 02:12:12.093343 26509 tablet_service.cc:1514] Processing DeleteTablet for tablet e866b4f1b338419ab1025287b2aaac5e with delete_type TABLET_DATA_TOMBSTONED (TS b2835e59d40a48aea53c5dedb3bd1e9e not found in new config with opid_index 103) from {username='slave'} at 127.0.0.1:34002
I20240503 02:12:12.108122 28379 tablet_replica.cc:330] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e: stopping tablet replica
I20240503 02:12:12.113495 28379 raft_consensus.cc:2229] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:12.115044 28379 pending_rounds.cc:62] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e: Trying to abort 1 pending ops.
I20240503 02:12:12.116276 28379 pending_rounds.cc:68] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e: Aborting op as it isn't in flight: id { term: 1 index: 102 } timestamp: 7023420749809479680 op_type: WRITE_OP write_request { tablet_id: "e866b4f1b338419ab1025287b2aaac5e" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000A\222\250B\377\013\357\314\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""0b855ad2401b2dcd" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023420749674315776 authz_token { token_data: "\010\227\222\321\261\006\"3\n\005slave\022*\n 02e35d1522244be29c80c4b1a91bff24\020\001\030\001 \001(\001" signature: "<redacted>""gK8\263\343\364s\224\205}\375\354\001z\215\247\211:\345\347\331\352S\206\035\203W$\n\336\\\351^\237\217\177-\374\3642{\374\306;\226<\255\350\002\330\270\264\237T\231\237\'\272\377X\355\034\322\312" signing_key_seq_num: 0 } } request_id { client_id: "0e1123653a054b70aa1efd2a0976dfdb" seq_no: 293 first_incomplete_seq_no: 293 attempt_no: 0 }
I20240503 02:12:12.118733 28379 raft_consensus.cc:2258] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:12.127842 28379 ts_tablet_manager.cc:1875] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:12.184132 28379 ts_tablet_manager.cc:1888] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.102
I20240503 02:12:12.235284 28379 log.cc:1196] T e866b4f1b338419ab1025287b2aaac5e P b2835e59d40a48aea53c5dedb3bd1e9e: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/wals/e866b4f1b338419ab1025287b2aaac5e
I20240503 02:12:12.236528 25854 catalog_manager.cc:4830] TS b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347): tablet e866b4f1b338419ab1025287b2aaac5e (table rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]) successfully deleted
I20240503 02:12:12.346472 28323 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap replayed 1/1 log segments. Stats: ops{read=150 overwritten=0 applied=150 ignored=0} inserts{seen=781 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:12.352541 28323 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221: Bootstrap complete.
I20240503 02:12:12.353014 28323 ts_tablet_manager.cc:1367] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221: Time spent bootstrapping tablet: real 1.119s user 0.234s sys 0.065s
I20240503 02:12:12.359525 28323 raft_consensus.cc:348] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: true } }
I20240503 02:12:12.359835 28323 raft_consensus.cc:729] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3bb103b2fd2344e7b55caf7389acf221, State: Initialized, Role: LEARNER
I20240503 02:12:12.360105 28323 consensus_queue.cc:260] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 150, Last appended: 1.150, Last appended by leader: 150, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: true } }
I20240503 02:12:12.360832 28323 ts_tablet_manager.cc:1398] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20240503 02:12:12.449010 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 112128us
I20240503 02:12:12.489647 28357 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a: Bootstrap replayed 1/1 log segments. Stats: ops{read=110 overwritten=0 applied=110 ignored=0} inserts{seen=741 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:12.508498 28357 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a: Bootstrap complete.
I20240503 02:12:12.508898 28357 ts_tablet_manager.cc:1367] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a: Time spent bootstrapping tablet: real 0.562s user 0.137s sys 0.035s
I20240503 02:12:12.509735 28357 raft_consensus.cc:348] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:12.510133 28357 raft_consensus.cc:729] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0a801f0897884f108cdb89434b18628a, State: Initialized, Role: LEARNER
I20240503 02:12:12.510443 28357 consensus_queue.cc:260] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 110, Last appended: 1.110, Last appended by leader: 110, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:12.535948 26159 tablet_copy_service.cc:342] P 2cf1cac003b44643a2d4de1204aacf0f: Request end of tablet copy session 3bb103b2fd2344e7b55caf7389acf221-041ffbedf2c04dbea4661b4c33f78ac1 received from {username='slave'} at 127.23.104.193:35363
I20240503 02:12:12.536119 26159 tablet_copy_service.cc:433] P 2cf1cac003b44643a2d4de1204aacf0f: ending tablet copy session 3bb103b2fd2344e7b55caf7389acf221-041ffbedf2c04dbea4661b4c33f78ac1 on tablet 041ffbedf2c04dbea4661b4c33f78ac1 with peer 3bb103b2fd2344e7b55caf7389acf221
I20240503 02:12:12.542171 26133 consensus_queue.cc:237] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 183, Committed index: 183, Last appended: 1.183, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:12.557710 26525 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 183. Preceding OpId from leader: term: 1 index: 184. (index mismatch)
I20240503 02:12:12.559173 28273 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 184, Last known committed idx: 183, Time since last communication: 0.000s
I20240503 02:12:12.563985 26007 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 183. Preceding OpId from leader: term: 1 index: 184. (index mismatch)
I20240503 02:12:12.569766 28357 ts_tablet_manager.cc:1398] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a: Time spent starting tablet: real 0.061s user 0.007s sys 0.007s
I20240503 02:12:12.573365 26289 tablet_copy_service.cc:342] P 7c384005cb3041a0bd70e163473b6acb: Request end of tablet copy session 0a801f0897884f108cdb89434b18628a-c27502976d474b938cce2235e5784f79 received from {username='slave'} at 127.23.104.199:38513
I20240503 02:12:12.573537 26289 tablet_copy_service.cc:433] P 7c384005cb3041a0bd70e163473b6acb: ending tablet copy session 0a801f0897884f108cdb89434b18628a-c27502976d474b938cce2235e5784f79 on tablet c27502976d474b938cce2235e5784f79 with peer 0a801f0897884f108cdb89434b18628a
I20240503 02:12:12.583597 26267 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 183. Preceding OpId from leader: term: 1 index: 184. (index mismatch)
I20240503 02:12:12.587178 28347 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 184, Last known committed idx: 183, Time since last communication: 0.000s
I20240503 02:12:12.589051 26399 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 183. Preceding OpId from leader: term: 1 index: 184. (index mismatch)
I20240503 02:12:12.592165 28347 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Committing config change with OpId 1.184: config changed from index -1 to 184, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:12.598359 26267 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.184: config changed from index -1 to 184, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:12.602741 27980 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 184, Last known committed idx: 183, Time since last communication: 0.000s
I20240503 02:12:12.608394 26397 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.184: config changed from index -1 to 184, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:12.611863 28347 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 184, Last known committed idx: 183, Time since last communication: 0.000s
I20240503 02:12:12.612360 25858 catalog_manager.cc:5484] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f reported cstate change: config changed from index -1 to 184, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:12.613744 26072 consensus_peers.cc:485] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 495c99a8850d4ec6b52a89bec172bab0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:12.614091 26008 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.184: config changed from index -1 to 184, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:12.665325 26524 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.184: config changed from index -1 to 184, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:12.667560 27594 raft_consensus.cc:1203] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.102->[1.103-1.114] Dedup: 1.110->[1.111-1.114]
I20240503 02:12:12.775971 26007 raft_consensus.cc:1203] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.146->[1.147-1.165] Dedup: 1.150->[1.151-1.165]
W20240503 02:12:12.802963 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 43839.1us
W20240503 02:12:12.913480 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 115139us
W20240503 02:12:12.936679 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 130404us
I20240503 02:12:12.994506 28273 raft_consensus.cc:1050] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f: attempting to promote NON_VOTER 3bb103b2fd2344e7b55caf7389acf221 to VOTER
I20240503 02:12:12.995499 28273 consensus_queue.cc:237] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 166, Committed index: 166, Last appended: 1.167, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false } }
I20240503 02:12:13.003647 26001 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 LEARNER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 02:12:13.005115 26269 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 166. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 02:12:13.006062 26659 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 02:12:13.006511 28405 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 166, Time since last communication: 0.000s
I20240503 02:12:13.006945 28405 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 166, Time since last communication: 0.000s
I20240503 02:12:13.007290 28405 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 165, Time since last communication: 0.000s
I20240503 02:12:13.016356 28347 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Committing config change with OpId 1.168: config changed from index 147 to 168, 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false } } }
I20240503 02:12:13.018890 26397 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 02:12:13.021605 26656 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index 147 to 168, 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false } } }
I20240503 02:12:13.027489 28273 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 166, Time since last communication: 0.000s
I20240503 02:12:13.027531 26008 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index 147 to 168, 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false } } }
I20240503 02:12:13.027952 25866 catalog_manager.cc:5484] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f reported cstate change: config changed from index 147 to 168, 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:13.030125 26268 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index 147 to 168, 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false } } }
I20240503 02:12:13.032796 28357 ts_tablet_manager.cc:902] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a: Initiating tablet copy from peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:12:13.036283 26397 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index 147 to 168, 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false } } }
I20240503 02:12:13.045482 26528 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 166. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 02:12:13.052776 28273 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 165, Time since last communication: 0.000s
I20240503 02:12:13.065616 26527 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index 147 to 168, 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false } } }
W20240503 02:12:13.104923 26204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 70215.8us
I20240503 02:12:13.110888 28334 raft_consensus.cc:1050] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb: attempting to promote NON_VOTER 0a801f0897884f108cdb89434b18628a to VOTER
I20240503 02:12:13.118793 28380 raft_consensus.cc:1050] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb: attempting to promote NON_VOTER 0a801f0897884f108cdb89434b18628a to VOTER
I20240503 02:12:13.128775 26136 consensus_queue.cc:237] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 170, Committed index: 170, Last appended: 1.170, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false } }
I20240503 02:12:13.133023 26656 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 02:12:13.133399 26004 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 02:12:13.133657 28273 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 170, Time since last communication: 0.000s
I20240503 02:12:13.137830 28273 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 170, Time since last communication: 0.000s
I20240503 02:12:13.141499 26527 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 02:12:13.142956 28404 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 170, Time since last communication: 0.000s
I20240503 02:12:13.146811 26392 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 169. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 02:12:13.146541 27980 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Committing config change with OpId 1.171: config changed from index 168 to 171, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false } } }
I20240503 02:12:13.148507 28335 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 168, Time since last communication: 0.000s
I20240503 02:12:13.151896 25851 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 041ffbedf2c04dbea4661b4c33f78ac1 with cas_config_opid_index 168: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:13.153412 26392 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 168 to 171, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false } } }
I20240503 02:12:13.153571 26656 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 168 to 171, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false } } }
I20240503 02:12:13.159668 25866 catalog_manager.cc:5484] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 reported cstate change: config changed from index 168 to 171, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false } } }
I20240503 02:12:13.141533 28334 consensus_queue.cc:237] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 1.116, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:13.161310 26524 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 168 to 171, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false } } }
W20240503 02:12:13.164461 28380 raft_consensus.cc:1054] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb: Unable to promote non-voter 0a801f0897884f108cdb89434b18628a: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }.
Pending config: opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:13.166000 26001 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
I20240503 02:12:13.167100 28225 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20240503 02:12:13.172955 27593 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
I20240503 02:12:13.174225 28348 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
W20240503 02:12:13.177624 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 61027.1us
I20240503 02:12:13.177917 26001 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 168 to 171, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false } } }
I20240503 02:12:13.180143 26392 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
I20240503 02:12:13.180931 28348 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20240503 02:12:13.191179 26247 tablet_service.cc:1514] Processing DeleteTablet for tablet 041ffbedf2c04dbea4661b4c33f78ac1 with delete_type TABLET_DATA_TOMBSTONED (TS 7c384005cb3041a0bd70e163473b6acb not found in new config with opid_index 171) from {username='slave'} at 127.0.0.1:37992
I20240503 02:12:13.191488 26659 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 115. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
I20240503 02:12:13.197554 26524 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
I20240503 02:12:13.206549 26397 raft_consensus.cc:1203] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.169->[1.170-1.170] Dedup: 1.170->[]
I20240503 02:12:13.214586 28348 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20240503 02:12:13.217156 28425 tablet_replica.cc:330] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb: stopping tablet replica
I20240503 02:12:13.217384 28425 raft_consensus.cc:2229] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:13.275135 28357 tablet_copy_client.cc:308] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:39753
W20240503 02:12:13.305191 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 46585.7us
I20240503 02:12:13.310896 28425 raft_consensus.cc:2258] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:13.316071 28334 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 114, Time since last communication: 0.084s
I20240503 02:12:13.327935 28425 ts_tablet_manager.cc:1875] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:13.345216 26399 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index 103 to 117, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:13.345754 26008 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index 103 to 117, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:13.345019 28380 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Committing config change with OpId 1.117: config changed from index 103 to 117, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:13.347947 27594 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index 103 to 117, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:13.358750 28425 ts_tablet_manager.cc:1888] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.170
I20240503 02:12:13.358887 28425 log.cc:1196] T 041ffbedf2c04dbea4661b4c33f78ac1 P 7c384005cb3041a0bd70e163473b6acb: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/041ffbedf2c04dbea4661b4c33f78ac1
I20240503 02:12:13.361130 25866 catalog_manager.cc:5484] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: config changed from index 103 to 117, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:13.364049 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 74127.5us
I20240503 02:12:13.366067 25853 catalog_manager.cc:4830] TS 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999): tablet 041ffbedf2c04dbea4661b4c33f78ac1 (table rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]) successfully deleted
I20240503 02:12:13.392939 26527 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index 103 to 117, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:13.412204 26268 consensus_queue.cc:237] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 136, Committed index: 136, Last appended: 1.136, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:13.413244 26656 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index 103 to 117, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:13.414685 26202 consensus_peers.cc:485] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21e7ddb56be1499b94920ebd252595ce. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:13.414851 26001 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 136. Preceding OpId from leader: term: 1 index: 137. (index mismatch)
I20240503 02:12:13.415258 28225 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 137, Last known committed idx: 136, Time since last communication: 0.000s
I20240503 02:12:13.415640 26134 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 136. Preceding OpId from leader: term: 1 index: 137. (index mismatch)
I20240503 02:12:13.416054 28225 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 137, Last known committed idx: 136, Time since last communication: 0.000s
I20240503 02:12:13.418629 26653 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 136. Preceding OpId from leader: term: 1 index: 137. (index mismatch)
I20240503 02:12:13.419389 26523 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 136. Preceding OpId from leader: term: 1 index: 137. (index mismatch)
I20240503 02:12:13.419183 28225 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Committing config change with OpId 1.137: config changed from index -1 to 137, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:13.419684 26134 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.137: config changed from index -1 to 137, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:13.427743 25858 catalog_manager.cc:5484] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f reported cstate change: config changed from index -1 to 137, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:13.429873 26001 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.137: config changed from index -1 to 137, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:13.435806 28406 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 137, Last known committed idx: 136, Time since last communication: 0.000s
I20240503 02:12:13.454071 28406 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 137, Last known committed idx: 136, Time since last communication: 0.000s
I20240503 02:12:13.468096 26264 consensus_queue.cc:237] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 1.119, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:13.473032 26005 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 119. Preceding OpId from leader: term: 1 index: 120. (index mismatch)
I20240503 02:12:13.475055 28406 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 118, Time since last communication: 0.000s
I20240503 02:12:13.478588 26656 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.137: config changed from index -1 to 137, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:13.487622 26528 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.137: config changed from index -1 to 137, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:13.505086 26159 tablet_copy_service.cc:140] P 2cf1cac003b44643a2d4de1204aacf0f: Received BeginTabletCopySession request for tablet 495c99a8850d4ec6b52a89bec172bab0 from peer 0a801f0897884f108cdb89434b18628a ({username='slave'} at 127.23.104.199:43229)
I20240503 02:12:13.505254 26159 tablet_copy_service.cc:161] P 2cf1cac003b44643a2d4de1204aacf0f: Beginning new tablet copy session on tablet 495c99a8850d4ec6b52a89bec172bab0 from peer 0a801f0897884f108cdb89434b18628a at {username='slave'} at 127.23.104.199:43229: session id = 0a801f0897884f108cdb89434b18628a-495c99a8850d4ec6b52a89bec172bab0
I20240503 02:12:13.507170 26659 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 120. (index mismatch)
I20240503 02:12:13.508767 26159 tablet_copy_source_session.cc:205] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:13.514238 28357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495c99a8850d4ec6b52a89bec172bab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:13.519599 26525 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 120. (index mismatch)
I20240503 02:12:13.519580 28406 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 117, Time since last communication: 0.000s
I20240503 02:12:13.521718 28357 tablet_copy_client.cc:783] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:13.522398 28357 tablet_copy_client.cc:647] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:13.523131 28429 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 117, Time since last communication: 0.000s
I20240503 02:12:13.537297 28429 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Committing config change with OpId 1.120: config changed from index 117 to 120, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:13.538120 28357 tablet_copy_client.cc:523] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:13.546228 26653 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.120: config changed from index 117 to 120, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:13.555554 26003 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.120: config changed from index 117 to 120, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:13.564159 25858 catalog_manager.cc:5484] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 reported cstate change: config changed from index 117 to 120, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:13.566112 28357 tablet_bootstrap.cc:492] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a: Bootstrap starting.
I20240503 02:12:13.571914 25853 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c27502976d474b938cce2235e5784f79 with cas_config_opid_index 117: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:13.584962 26378 tablet_service.cc:1514] Processing DeleteTablet for tablet c27502976d474b938cce2235e5784f79 with delete_type TABLET_DATA_TOMBSTONED (TS cff7b7a19fc64db5849391b1d8e10cce not found in new config with opid_index 120) from {username='slave'} at 127.0.0.1:48358
I20240503 02:12:13.616474 27593 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 119. Preceding OpId from leader: term: 1 index: 120. (index mismatch)
I20240503 02:12:13.617439 28443 tablet_replica.cc:330] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce: stopping tablet replica
I20240503 02:12:13.617930 28443 raft_consensus.cc:2229] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:13.618219 28443 pending_rounds.cc:62] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce: Trying to abort 1 pending ops.
I20240503 02:12:13.618361 28443 pending_rounds.cc:68] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce: Aborting op as it isn't in flight: id { term: 1 index: 119 } timestamp: 7023420757849382912 op_type: WRITE_OP write_request { tablet_id: "c27502976d474b938cce2235e5784f79" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\210XkN\255\322\0351\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""f8f13ad64a9ba99d" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023420757780316160 authz_token { token_data: "\010\227\222\321\261\006\"3\n\005slave\022*\n 02e35d1522244be29c80c4b1a91bff24\020\001\030\001 \001(\001" signature: "<redacted>""gK8\263\343\364s\224\205}\375\354\001z\215\247\211:\345\347\331\352S\206\035\203W$\n\336\\\351^\237\217\177-\374\3642{\374\306;\226<\255\350\002\330\270\264\237T\231\237\'\272\377X\355\034\322\312" signing_key_seq_num: 0 } } request_id { client_id: "0e1123653a054b70aa1efd2a0976dfdb" seq_no: 320 first_incomplete_seq_no: 320 attempt_no: 0 }
I20240503 02:12:13.618989 28443 raft_consensus.cc:2258] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:13.647390 28429 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 118, Time since last communication: 0.000s
I20240503 02:12:13.649107 26524 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.120: config changed from index 117 to 120, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:13.658412 27592 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.120: config changed from index 117 to 120, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:13.845826 28453 ts_tablet_manager.cc:902] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a: Initiating tablet copy from peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999)
I20240503 02:12:14.084636 28443 ts_tablet_manager.cc:1875] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:14.420017 28453 tablet_copy_client.cc:308] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:45999
I20240503 02:12:14.422235 26289 tablet_copy_service.cc:140] P 7c384005cb3041a0bd70e163473b6acb: Received BeginTabletCopySession request for tablet 21e7ddb56be1499b94920ebd252595ce from peer 0a801f0897884f108cdb89434b18628a ({username='slave'} at 127.23.104.199:38513)
I20240503 02:12:14.422412 26289 tablet_copy_service.cc:161] P 7c384005cb3041a0bd70e163473b6acb: Beginning new tablet copy session on tablet 21e7ddb56be1499b94920ebd252595ce from peer 0a801f0897884f108cdb89434b18628a at {username='slave'} at 127.23.104.199:38513: session id = 0a801f0897884f108cdb89434b18628a-21e7ddb56be1499b94920ebd252595ce
I20240503 02:12:14.425277 26289 tablet_copy_source_session.cc:205] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:14.426931 28453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e7ddb56be1499b94920ebd252595ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:14.436060 28453 tablet_copy_client.cc:783] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:14.436345 28453 tablet_copy_client.cc:647] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:14.439347 28453 tablet_copy_client.cc:523] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:14.452769 28453 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a: Bootstrap starting.
I20240503 02:12:14.724104 28443 ts_tablet_manager.cc:1888] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.119
I20240503 02:12:14.746482 28443 log.cc:1196] T c27502976d474b938cce2235e5784f79 P cff7b7a19fc64db5849391b1d8e10cce: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/c27502976d474b938cce2235e5784f79
I20240503 02:12:14.765030 28443 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48358 (request call id 15) took 1180 ms. Trace:
I20240503 02:12:14.767915 28443 rpcz_store.cc:276] 0503 02:12:13.584715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:13.584761 (+ 46us) service_pool.cc:225] Handling call
0503 02:12:13.617252 (+ 32491us) ts_tablet_manager.cc:1120] Deleting tablet c27502976d474b938cce2235e5784f79
0503 02:12:13.617264 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:14.503241 (+885977us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:14.723847 (+220606us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:14.764977 (+ 41130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":31930,"thread_start_us":37797,"threads_started":1}
I20240503 02:12:14.896865 28357 tablet_bootstrap.cc:492] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a: Bootstrap replayed 1/1 log segments. Stats: ops{read=189 overwritten=0 applied=189 ignored=0} inserts{seen=861 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:14.897392 28357 tablet_bootstrap.cc:492] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a: Bootstrap complete.
I20240503 02:12:14.899295 28357 ts_tablet_manager.cc:1367] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a: Time spent bootstrapping tablet: real 1.333s user 0.271s sys 0.014s
I20240503 02:12:14.901247 28357 raft_consensus.cc:348] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:14.902926 28357 raft_consensus.cc:729] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0a801f0897884f108cdb89434b18628a, State: Initialized, Role: LEARNER
I20240503 02:12:14.907043 28357 consensus_queue.cc:260] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 189, Last appended: 1.189, Last appended by leader: 189, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:14.913383 28357 ts_tablet_manager.cc:1398] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240503 02:12:14.932545 25853 catalog_manager.cc:4830] TS cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675): tablet c27502976d474b938cce2235e5784f79 (table rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]) successfully deleted
I20240503 02:12:14.969131 26159 tablet_copy_service.cc:342] P 2cf1cac003b44643a2d4de1204aacf0f: Request end of tablet copy session 0a801f0897884f108cdb89434b18628a-495c99a8850d4ec6b52a89bec172bab0 received from {username='slave'} at 127.23.104.199:43229
I20240503 02:12:14.969321 26159 tablet_copy_service.cc:433] P 2cf1cac003b44643a2d4de1204aacf0f: ending tablet copy session 0a801f0897884f108cdb89434b18628a-495c99a8850d4ec6b52a89bec172bab0 on tablet 495c99a8850d4ec6b52a89bec172bab0 with peer 0a801f0897884f108cdb89434b18628a
W20240503 02:12:14.989359 25941 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 145708us
I20240503 02:12:15.025167 27592 raft_consensus.cc:1203] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.183->[1.184-1.193] Dedup: 1.189->[1.190-1.193]
W20240503 02:12:15.032878 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 59245.1us
I20240503 02:12:15.178511 28453 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a: Bootstrap replayed 1/1 log segments. Stats: ops{read=152 overwritten=0 applied=151 ignored=0} inserts{seen=782 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:12:15.179359 28453 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a: Bootstrap complete.
I20240503 02:12:15.180626 28453 ts_tablet_manager.cc:1367] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a: Time spent bootstrapping tablet: real 0.728s user 0.201s sys 0.059s
I20240503 02:12:15.181584 28453 raft_consensus.cc:348] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:15.183817 28453 raft_consensus.cc:729] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0a801f0897884f108cdb89434b18628a, State: Initialized, Role: LEARNER
I20240503 02:12:15.184095 28453 consensus_queue.cc:260] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 151, Last appended: 1.152, Last appended by leader: 152, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:15.184911 28453 ts_tablet_manager.cc:1398] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:12:15.188611 26289 tablet_copy_service.cc:342] P 7c384005cb3041a0bd70e163473b6acb: Request end of tablet copy session 0a801f0897884f108cdb89434b18628a-21e7ddb56be1499b94920ebd252595ce received from {username='slave'} at 127.23.104.199:38513
I20240503 02:12:15.188732 26289 tablet_copy_service.cc:433] P 7c384005cb3041a0bd70e163473b6acb: ending tablet copy session 0a801f0897884f108cdb89434b18628a-21e7ddb56be1499b94920ebd252595ce on tablet 21e7ddb56be1499b94920ebd252595ce with peer 0a801f0897884f108cdb89434b18628a
I20240503 02:12:15.195447 28404 raft_consensus.cc:1050] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f: attempting to promote NON_VOTER 0a801f0897884f108cdb89434b18628a to VOTER
I20240503 02:12:15.196861 28404 consensus_queue.cc:237] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 193, Committed index: 193, Last appended: 1.194, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:15.201402 26527 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 1 index: 195. (index mismatch)
I20240503 02:12:15.203006 27592 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 1 index: 195. (index mismatch)
I20240503 02:12:15.202936 28273 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 193, Time since last communication: 0.000s
I20240503 02:12:15.203893 28335 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 193, Time since last communication: 0.000s
I20240503 02:12:15.204430 26393 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 1 index: 195. (index mismatch)
I20240503 02:12:15.205250 28335 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 193, Time since last communication: 0.000s
I20240503 02:12:15.209656 28404 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Committing config change with OpId 1.195: config changed from index 184 to 195, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:15.220175 26267 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 1 index: 195. (index mismatch)
I20240503 02:12:15.221827 28335 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 193, Time since last communication: 0.000s
I20240503 02:12:15.225266 26267 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.195: config changed from index 184 to 195, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:15.244657 25866 catalog_manager.cc:5484] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb reported cstate change: config changed from index 184 to 195, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:15.270537 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 61200.1us
I20240503 02:12:15.361750 26527 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.195: config changed from index 184 to 195, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:15.385488 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 157868us
I20240503 02:12:15.430670 27593 raft_consensus.cc:1203] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.136->[1.137-1.164] Dedup: 1.152->[1.153-1.164]
W20240503 02:12:15.521507 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 53997.2us
W20240503 02:12:15.562752 26462 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 188723us
I20240503 02:12:15.593590 26007 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 1 index: 195. (index mismatch)
I20240503 02:12:15.594849 26399 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.195: config changed from index 184 to 195, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:15.608654 27980 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 193, Time since last communication: 0.000s
I20240503 02:12:15.615878 27593 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.195: config changed from index 184 to 195, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:15.623978 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 101025us
W20240503 02:12:15.646291 25840 kernel_stack_watchdog.cc:198] Thread 28517 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/consensus/log.cc:407 for 210ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f1de4c82980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f1de18f0d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f1de19117ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f1de190d3ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f1de190b84f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7f1ded06e1de kudu::consensus::PeerMessageQueue::NotifyObserversOfCommitIndexChange() at ??:0
@ 0x7f1ded05c9cc kudu::consensus::PeerMessageQueue::ResponseFromPeer() at ??:0
@ 0x7f1ded05a7a4 kudu::consensus::PeerMessageQueue::LocalPeerAppendFinished() at ??:0
@ 0x7f1ded0724da kudu::consensus::PeerMessageQueue::AppendOperations()::$_1::operator()() at ??:0
@ 0x7f1ded072242 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f1df4e1f17e std::function<>::operator()() at ??:0
@ 0x7f1ded0bf57e kudu::consensus::LogCache::LogCallback() at ??:0
@ 0x7f1ded0c435f kudu::consensus::LogCache::AppendOperations()::$_0::operator()() at ??:0
W20240503 02:12:15.805083 26581 kernel_stack_watchdog.cc:198] Thread 26658 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 286ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f3a1c61f980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f3a1928dd11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f3a192ae7ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f3a192aa3ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f3a192a884f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7f3a25e1465a kudu::tablet::OpDriver::ExecuteAsync() at ??:0
@ 0x7f3a25dc1ffb kudu::tablet::TabletReplica::StartFollowerOp() at ??:0
@ 0x7f3a24a9e1e2 kudu::consensus::RaftConsensus::StartFollowerOpUnlocked() at ??:0
@ 0x7f3a24abcd39 kudu::consensus::RaftConsensus::UpdateReplica() at ??:0
@ 0x7f3a24aba678 kudu::consensus::RaftConsensus::Update() at ??:0
@ 0x7f3a2879a10e kudu::tserver::ConsensusServiceImpl::UpdateConsensus() at ??:0
@ 0x7f3a22e9eb57 kudu::consensus::ConsensusServiceIf::ConsensusServiceIf()::$_1::operator()() at ??:0
@ 0x7f3a22e9e886 std::_Function_handler<>::_M_invoke() at ??:0
W20240503 02:12:15.895649 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 148045us
I20240503 02:12:15.905040 26007 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.195: config changed from index 184 to 195, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:15.928766 26658 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.004s sys 0.001s
W20240503 02:12:15.928875 26658 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.610s user 0.004s sys 0.001s
W20240503 02:12:15.928931 28334 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.005s sys 0.002s
W20240503 02:12:15.935271 28334 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.616s user 0.005s sys 0.002s
I20240503 02:12:15.936815 28334 raft_consensus.cc:1050] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb: attempting to promote NON_VOTER 0a801f0897884f108cdb89434b18628a to VOTER
W20240503 02:12:15.938805 28517 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.093s sys 0.173s
W20240503 02:12:15.938901 28517 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.631s user 0.093s sys 0.173s
W20240503 02:12:15.946396 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 49648.6us
W20240503 02:12:15.947537 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 51627.3us
I20240503 02:12:15.994601 28334 consensus_queue.cc:237] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 164, Committed index: 164, Last appended: 1.164, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
W20240503 02:12:16.000036 25941 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 314396us
W20240503 02:12:16.014182 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 66439.5us
W20240503 02:12:16.023295 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 76716.7us
I20240503 02:12:16.031852 27593 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 164. Preceding OpId from leader: term: 1 index: 165. (index mismatch)
I20240503 02:12:16.033197 26004 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 164. Preceding OpId from leader: term: 1 index: 165. (index mismatch)
I20240503 02:12:16.040992 26656 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 164. Preceding OpId from leader: term: 1 index: 165. (index mismatch)
I20240503 02:12:16.042234 28293 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 164, Time since last communication: 0.000s
I20240503 02:12:16.042711 28293 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 164, Time since last communication: 0.000s
I20240503 02:12:16.047896 26136 consensus_queue.cc:237] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 196, Committed index: 196, Last appended: 1.196, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:16.052238 26134 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 164. Preceding OpId from leader: term: 1 index: 165. (index mismatch)
I20240503 02:12:16.053679 26005 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 196. Preceding OpId from leader: term: 1 index: 197. (index mismatch)
I20240503 02:12:16.054080 26269 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 196. Preceding OpId from leader: term: 1 index: 197. (index mismatch)
I20240503 02:12:16.054240 28515 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 164, Time since last communication: 0.000s
I20240503 02:12:16.054765 27593 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 196. Preceding OpId from leader: term: 1 index: 197. (index mismatch)
I20240503 02:12:16.055274 28347 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 197, Last known committed idx: 196, Time since last communication: 0.000s
I20240503 02:12:16.055727 28347 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 197, Last known committed idx: 196, Time since last communication: 0.000s
I20240503 02:12:16.058753 28347 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 197, Last known committed idx: 195, Time since last communication: 0.000s
I20240503 02:12:16.059185 28347 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Committing config change with OpId 1.197: config changed from index 195 to 197, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New config: { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.065220 28293 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 164, Time since last communication: 0.000s
I20240503 02:12:16.066362 25851 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 495c99a8850d4ec6b52a89bec172bab0 with cas_config_opid_index 195: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:16.067363 26265 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.197: config changed from index 195 to 197, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New config: { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.068733 27593 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.197: config changed from index 195 to 197, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New config: { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.072448 26399 raft_consensus.cc:1261] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 196. Preceding OpId from leader: term: 1 index: 197. (index mismatch)
I20240503 02:12:16.073168 26005 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.197: config changed from index 195 to 197, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New config: { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.077664 25861 catalog_manager.cc:5484] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a reported cstate change: config changed from index 195 to 197, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.077611 28293 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Committing config change with OpId 1.165: config changed from index 137 to 165, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.080647 26136 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.165: config changed from index 137 to 165, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.087563 28335 consensus_queue.cc:1035] T 495c99a8850d4ec6b52a89bec172bab0 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 197, Last known committed idx: 196, Time since last communication: 0.000s
I20240503 02:12:16.098089 26397 raft_consensus.cc:2932] T 495c99a8850d4ec6b52a89bec172bab0 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.197: config changed from index 195 to 197, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New config: { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.110926 26656 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.165: config changed from index 137 to 165, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.117405 25862 catalog_manager.cc:5484] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 reported cstate change: config changed from index 137 to 165, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.160629 26527 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 164. Preceding OpId from leader: term: 1 index: 165. (index mismatch)
W20240503 02:12:16.165838 25941 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 46125.9us
W20240503 02:12:16.364531 26191 kernel_stack_watchdog.cc:198] Thread 28334 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 263ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1553b1980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7fa15201fd11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7fa1520407ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fa15203c3ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fa15203a84f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7fa15d76ad65 kudu::consensus::Peer::SignalRequest() at ??:0
@ 0x7fa15d7fe47d kudu::consensus::PeerManager::SignalRequest() at ??:0
@ 0x7fa15d864920 kudu::consensus::RaftConsensus::BulkChangeConfig() at ??:0
@ 0x7fa15d8498b8 kudu::consensus::RaftConsensus::ChangeConfig() at ??:0
@ 0x7fa15d84ae3e kudu::consensus::RaftConsensus::TryPromoteNonVoterTask() at ??:0
@ 0x7fa15d87b465 kudu::consensus::RaftConsensus::NotifyPeerToPromote()::$_7::operator()() at ??:0
@ 0x7fa15d87b1ce std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
W20240503 02:12:16.410138 26451 kernel_stack_watchdog.cc:198] Thread 26464 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 155ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f0c4b855980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f0c484c3d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f0c484e47ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f0c484e03ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f0c484de84f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7f0c53c119e8 kudu::consensus::Peer::ProcessResponse() at ??:0
@ 0x7f0c53c18061 kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
@ 0x7f0c53c17e6e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f0c4f1158cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f0c4f1154ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f0c4f1014e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
I20240503 02:12:16.431943 26509 tablet_service.cc:1514] Processing DeleteTablet for tablet 495c99a8850d4ec6b52a89bec172bab0 with delete_type TABLET_DATA_TOMBSTONED (TS b2835e59d40a48aea53c5dedb3bd1e9e not found in new config with opid_index 197) from {username='slave'} at 127.0.0.1:34002
W20240503 02:12:16.431943 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 258408us
I20240503 02:12:16.449801 28521 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 164, Time since last communication: 0.000s
I20240503 02:12:16.468693 27593 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.165: config changed from index 137 to 165, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:16.492986 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 48610.7us
I20240503 02:12:16.502045 26008 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.165: config changed from index 137 to 165, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.538411 26525 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.165: config changed from index 137 to 165, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.549034 28563 tablet_replica.cc:330] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e: stopping tablet replica
I20240503 02:12:16.549260 28563 raft_consensus.cc:2229] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:16.549472 28563 raft_consensus.cc:2258] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:16.552675 28563 ts_tablet_manager.cc:1875] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:16.614214 28549 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55512 (ReqId={client: aa5d1cf5815349fd826312d67e9703d7, seq_no=535, attempt_no=0}) took 1038 ms. Trace:
I20240503 02:12:16.614393 28549 rpcz_store.cc:276] 0503 02:12:15.575349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:15.585166 (+ 9817us) service_pool.cc:225] Handling call
0503 02:12:16.613715 (+1028549us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:15.585783 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 21e7ddb56be1499b94920ebd252595ce
0503 02:12:15.585909 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:15.585916 (+ 7us) write_op.cc:435] Acquired schema lock
0503 02:12:15.585921 (+ 5us) tablet.cc:660] Decoding operations
0503 02:12:15.586214 (+ 293us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:15.586229 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:15.586236 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:15.586289 (+ 53us) tablet.cc:699] Row locks acquired
0503 02:12:15.586294 (+ 5us) write_op.cc:260] PREPARE: finished
0503 02:12:15.586337 (+ 43us) write_op.cc:270] Start()
0503 02:12:15.586369 (+ 32us) write_op.cc:276] Timestamp: P: 1714702335586293 usec, L: 0
0503 02:12:15.586375 (+ 6us) op_driver.cc:331] REPLICATION: starting
0503 02:12:16.499350 (+912975us) log.cc:842] Serialized 357 byte log entry
0503 02:12:16.547091 (+ 47741us) spinlock_profiling.cc:243] Waited 913 ms on lock 0x616000083254. stack: 00007fa151ffc756 00007fa151ffc510 00007fa15102f48d 0000000000955804 00000000009557cd 000000000095519a 00007fa15d841162 00007fa15eba9e32 00007fa15eba8959 00007fa15ebb086b 00007fa15ebb06e8 0000000000a60bcc 00007fa152042b74 00007fa15204619b 00007fa152046018 0000000000a60bcc
0503 02:12:16.611420 (+ 64329us) op_driver.cc:444] REPLICATION: finished
0503 02:12:16.611533 (+ 113us) write_op.cc:301] APPLY: starting
0503 02:12:16.611547 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:16.611586 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:16.611588 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:16.611642 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:16.611672 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:16.611679 (+ 7us) write_op.cc:312] APPLY: finished
0503 02:12:16.613541 (+ 1862us) log.cc:842] Serialized 23 byte log entry
0503 02:12:16.613619 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:16.613635 (+ 16us) write_op.cc:454] Released schema lock
0503 02:12:16.613640 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":60,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":3946,"prepare.run_wall_time_us":961372,"raft.queue_time_us":283,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":185,"replication_time_us":1025017,"spinlock_wait_cycles":2099254656}]]}
W20240503 02:12:16.725191 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 160798us
I20240503 02:12:16.799317 26269 consensus_queue.cc:237] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 169, Committed index: 169, Last appended: 1.169, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:16.801702 27594 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 169. Preceding OpId from leader: term: 1 index: 170. (index mismatch)
I20240503 02:12:16.802711 28293 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 170, Last known committed idx: 169, Time since last communication: 0.000s
I20240503 02:12:16.804558 26004 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 169. Preceding OpId from leader: term: 1 index: 170. (index mismatch)
I20240503 02:12:16.806124 28575 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 170, Last known committed idx: 169, Time since last communication: 0.000s
I20240503 02:12:16.808840 28521 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Committing config change with OpId 1.170: config changed from index 165 to 170, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.816484 27594 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.170: config changed from index 165 to 170, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.819589 26006 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.170: config changed from index 165 to 170, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.822949 25859 catalog_manager.cc:5484] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a reported cstate change: config changed from index 165 to 170, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:16.841512 26658 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 169. Preceding OpId from leader: term: 1 index: 170. (index mismatch)
I20240503 02:12:16.842450 26524 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 166. Preceding OpId from leader: term: 1 index: 170. (index mismatch)
I20240503 02:12:16.842825 25853 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21e7ddb56be1499b94920ebd252595ce with cas_config_opid_index 165: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:16.851411 28521 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 170, Last known committed idx: 166, Time since last communication: 0.000s
I20240503 02:12:16.909710 26117 tablet_service.cc:1514] Processing DeleteTablet for tablet 21e7ddb56be1499b94920ebd252595ce with delete_type TABLET_DATA_TOMBSTONED (TS 2cf1cac003b44643a2d4de1204aacf0f not found in new config with opid_index 170) from {username='slave'} at 127.0.0.1:54956
I20240503 02:12:16.942938 26135 consensus_queue.cc:237] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 1.126, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:16.946275 26269 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 02:12:16.947022 26393 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 02:12:16.947688 28273 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240503 02:12:16.948091 28273 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240503 02:12:16.953012 28273 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Committing config change with OpId 1.127: config changed from index -1 to 127, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:16.956403 26399 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index -1 to 127, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:16.963390 25859 catalog_manager.cc:5484] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: config changed from index -1 to 127, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:16.972534 26269 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index -1 to 127, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
W20240503 02:12:16.974364 26072 consensus_peers.cc:485] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c12cc2e0fb09431e95baae55803f2184. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:16.974949 26008 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 02:12:16.975930 28537 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240503 02:12:16.978338 26006 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index -1 to 127, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
W20240503 02:12:17.004899 28601 log.cc:924] Time spent T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20240503 02:12:17.023201 28600 tablet_replica.cc:330] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f: stopping tablet replica
I20240503 02:12:17.047135 26527 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 02:12:17.023769 28600 raft_consensus.cc:2229] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.058857 28600 raft_consensus.cc:2258] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.061293 28537 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 123, Time since last communication: 0.000s
I20240503 02:12:17.065672 28521 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 170, Last known committed idx: 168, Time since last communication: 0.000s
I20240503 02:12:17.065433 26527 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index -1 to 127, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:17.071897 26523 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.170: config changed from index 165 to 170, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:17.075577 28563 ts_tablet_manager.cc:1888] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.196
I20240503 02:12:17.201031 28600 ts_tablet_manager.cc:1875] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:17.505249 28624 ts_tablet_manager.cc:902] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a: Initiating tablet copy from peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:12:18.287186 28600 ts_tablet_manager.cc:1888] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.169
I20240503 02:12:18.298341 28600 log.cc:1196] T 21e7ddb56be1499b94920ebd252595ce P 2cf1cac003b44643a2d4de1204aacf0f: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/21e7ddb56be1499b94920ebd252595ce
I20240503 02:12:18.299093 28600 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54956 (request call id 15) took 1389 ms. Trace:
I20240503 02:12:18.299234 28600 rpcz_store.cc:276] 0503 02:12:16.909206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:16.909264 (+ 58us) service_pool.cc:225] Handling call
0503 02:12:17.022330 (+113066us) ts_tablet_manager.cc:1120] Deleting tablet 21e7ddb56be1499b94920ebd252595ce
0503 02:12:17.022349 (+ 19us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:18.283218 (+1260869us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:18.287023 (+ 3805us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:18.299030 (+ 12007us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":112178,"thread_start_us":114812,"threads_started":1}
I20240503 02:12:18.301218 25851 catalog_manager.cc:4830] TS 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): tablet 21e7ddb56be1499b94920ebd252595ce (table rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]) successfully deleted
W20240503 02:12:18.320129 26202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 52256.8us
I20240503 02:12:18.326560 28624 tablet_copy_client.cc:308] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:39753
I20240503 02:12:18.330475 26159 tablet_copy_service.cc:140] P 2cf1cac003b44643a2d4de1204aacf0f: Received BeginTabletCopySession request for tablet c12cc2e0fb09431e95baae55803f2184 from peer 0a801f0897884f108cdb89434b18628a ({username='slave'} at 127.23.104.199:43229)
I20240503 02:12:18.330641 26159 tablet_copy_service.cc:161] P 2cf1cac003b44643a2d4de1204aacf0f: Beginning new tablet copy session on tablet c12cc2e0fb09431e95baae55803f2184 from peer 0a801f0897884f108cdb89434b18628a at {username='slave'} at 127.23.104.199:43229: session id = 0a801f0897884f108cdb89434b18628a-c12cc2e0fb09431e95baae55803f2184
I20240503 02:12:18.333704 26159 tablet_copy_source_session.cc:205] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:18.337654 28624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12cc2e0fb09431e95baae55803f2184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.350436 28624 tablet_copy_client.cc:783] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:18.350688 28624 tablet_copy_client.cc:647] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:18.356801 28624 tablet_copy_client.cc:523] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:18.379833 28624 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a: Bootstrap starting.
I20240503 02:12:17.079442 26653 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.170: config changed from index 165 to 170, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:18.856235 28619 raft_consensus.cc:482] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae8b30bcfa5b425782a95374787dd230)
I20240503 02:12:18.846870 28410 raft_consensus.cc:482] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae8b30bcfa5b425782a95374787dd230)
I20240503 02:12:18.846475 28667 raft_consensus.cc:482] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae8b30bcfa5b425782a95374787dd230)
I20240503 02:12:18.867529 28619 raft_consensus.cc:504] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:12:18.867862 28410 raft_consensus.cc:504] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:12:18.867983 28667 raft_consensus.cc:504] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:12:18.868708 28619 leader_election.cc:288] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143)
I20240503 02:12:18.868935 28667 leader_election.cc:288] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:12:18.869141 28410 leader_election.cc:288] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:12:18.869593 26393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 232 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:12:18.851130 28436 raft_consensus.cc:482] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae8b30bcfa5b425782a95374787dd230)
I20240503 02:12:18.869864 26393 raft_consensus.cc:2396] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2cf1cac003b44643a2d4de1204aacf0f for term 2 because replica has last-logged OpId of term: 1 index: 233, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 232.
I20240503 02:12:18.869915 28436 raft_consensus.cc:504] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:12:18.870162 26397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:12:18.870249 26139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:12:18.870411 26397 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 1.
I20240503 02:12:18.870468 26139 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 1.
I20240503 02:12:17.341639 26657 consensus_queue.cc:237] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 232, Committed index: 232, Last appended: 1.233, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:17.075695 28563 log.cc:1196] T 495c99a8850d4ec6b52a89bec172bab0 P b2835e59d40a48aea53c5dedb3bd1e9e: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/wals/495c99a8850d4ec6b52a89bec172bab0
I20240503 02:12:18.871672 26397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:12:18.871085 28436 leader_election.cc:288] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:12:18.902312 26393 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch)
I20240503 02:12:19.492775 28563 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34002 (request call id 14) took 3060 ms. Trace:
I20240503 02:12:18.882445 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:12:18.871410 26005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 232 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:12:19.492902 28563 rpcz_store.cc:276] 0503 02:12:16.431757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:16.431826 (+ 69us) service_pool.cc:225] Handling call
0503 02:12:16.548799 (+116973us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x60c00039d280. stack: 00007f0c484a0756 00007f0c484a0510 00007f0c474d348d 0000000000955804 00000000009557cd 000000000095519a 00007f0c5b9337f5 00007f0c5369fa30 00007f0c484c4499 0000000000a4d186 00007f0c484e47aa 00007f0c484e03ac 00007f0c484e54fe 00007f0c57a7b6b5 00007f0c579c7c6f 00007f0c51e01466
0503 02:12:16.548838 (+ 39us) ts_tablet_manager.cc:1120] Deleting tablet 495c99a8850d4ec6b52a89bec172bab0
0503 02:12:16.548853 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:16.638837 (+ 89984us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:17.075435 (+436598us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:19.492710 (+2417275us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4680448,"tablet-delete.queue_time_us":113926,"thread_start_us":114110,"threads_started":1}
I20240503 02:12:18.888626 26137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:12:18.871861 26654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:12:18.922116 26004 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch)
I20240503 02:12:18.824577 28624 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a: Bootstrap replayed 1/1 log segments. Stats: ops{read=132 overwritten=0 applied=132 ignored=0} inserts{seen=804 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:18.908326 26139 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 232. Preceding OpId from leader: term: 1 index: 234. (index mismatch)
I20240503 02:12:19.494329 28673 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 232, Time since last communication: 0.000s
I20240503 02:12:18.925834 26001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:12:19.494663 28624 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a: Bootstrap complete.
I20240503 02:12:19.494699 26001 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 1.
I20240503 02:12:18.930967 26003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:12:19.494949 28624 ts_tablet_manager.cc:1367] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a: Time spent bootstrapping tablet: real 1.115s user 0.164s sys 0.024s
I20240503 02:12:19.494957 26003 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c384005cb3041a0bd70e163473b6acb in term 1.
I20240503 02:12:19.495069 25854 catalog_manager.cc:4830] TS b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347): tablet 495c99a8850d4ec6b52a89bec172bab0 (table rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]) successfully deleted
I20240503 02:12:19.495672 28624 raft_consensus.cc:348] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:19.495962 28624 raft_consensus.cc:729] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0a801f0897884f108cdb89434b18628a, State: Initialized, Role: LEARNER
I20240503 02:12:19.496160 28624 consensus_queue.cc:260] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 132, Last appended: 1.132, Last appended by leader: 132, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:18.870975 26264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 232 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:12:19.496732 26264 raft_consensus.cc:2396] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2cf1cac003b44643a2d4de1204aacf0f for term 2 because replica has last-logged OpId of term: 1 index: 233, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 232.
I20240503 02:12:18.908576 28576 raft_consensus.cc:482] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7c384005cb3041a0bd70e163473b6acb)
I20240503 02:12:19.497035 28576 raft_consensus.cc:504] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:19.498438 28576 leader_election.cc:288] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279)
I20240503 02:12:18.922370 26265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:12:18.929100 26269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:12:19.498699 26265 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 1.
I20240503 02:12:18.871917 26202 leader_election.cc:302] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f, 7c384005cb3041a0bd70e163473b6acb, cff7b7a19fc64db5849391b1d8e10cce; no voters:
I20240503 02:12:19.498790 26527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e7ddb56be1499b94920ebd252595ce" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 2 candidate_status { last_received { term: 1 index: 170 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:12:18.908607 26653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.23.104.195:34683 (request call id 691) took 1829 ms. Trace:
W20240503 02:12:19.499045 26202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 627081us
I20240503 02:12:18.920730 26656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:12:19.498992 26653 rpcz_store.cc:276] 0503 02:12:17.078644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:17.078692 (+ 48us) service_pool.cc:225] Handling call
0503 02:12:17.078854 (+ 162us) raft_consensus.cc:1457] Updating replica for 1 ops
0503 02:12:17.079025 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 169
0503 02:12:17.079102 (+ 77us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0503 02:12:17.079320 (+ 218us) log.cc:842] Serialized 751 byte log entry
0503 02:12:17.079365 (+ 45us) raft_consensus.cc:1631] Marking committed up to 170
0503 02:12:18.870411 (+1791046us) log.cc:842] Serialized 15 byte log entry
0503 02:12:18.908488 (+ 38077us) raft_consensus.cc:1683] Filling consensus response to leader.
0503 02:12:18.908514 (+ 26us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0503 02:12:18.908521 (+ 7us) raft_consensus.cc:1671] finished
0503 02:12:18.908523 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0503 02:12:18.908569 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":28,"raft.run_cpu_time_us":41,"raft.run_wall_time_us":42,"thread_start_us":37881,"threads_started":1,"wal-append.queue_time_us":37594}
W20240503 02:12:19.083225 26191 kernel_stack_watchdog.cc:198] Thread 26202 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 102ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1553b1980 (unknown) at ??:0
@ 0x7fa1553ab9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7fa14f331904 google::LogMessage::Flush() at ??:0
@ 0x7fa14f331a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fa15d7d73f0 kudu::consensus::LeaderElection::CheckForDecision() at ??:0
@ 0x7fa15d7d8782 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7fa15d7db12a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7fa15d7daeee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fa158c718cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7fa158c714ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fa158c5d4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa158c5da2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa158bfa51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa158bf8f45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240503 02:12:19.500064 28667 raft_consensus.cc:2781] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:12:19.500190 28667 raft_consensus.cc:482] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae8b30bcfa5b425782a95374787dd230)
I20240503 02:12:19.500291 28667 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:19.049831 26657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 232 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:12:19.080343 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:12:19.503126 28667 raft_consensus.cc:504] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:12:19.503667 26265 raft_consensus.cc:1226] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 2 FOLLOWER]: Rejecting Update request from peer ae8b30bcfa5b425782a95374787dd230 for earlier term 1. Current term is 2. Ops: []
W20240503 02:12:19.025919 26592 consensus_peers.cc:485] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a864ac954e046eda8925f5daf4a0f1f. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:19.504901 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 479112us
W20240503 02:12:19.369738 26581 kernel_stack_watchdog.cc:198] Thread 26592 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 279ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f3a1c61f980 (unknown) at ??:0
@ 0x7f3a1c6199c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f3a1659f904 google::LogMessage::Flush() at ??:0
@ 0x7f3a1659fa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f3a249dc9ee kudu::consensus::Peer::ProcessResponseErrorUnlocked() at ??:0
@ 0x7f3a249dbadf kudu::consensus::Peer::ProcessResponse() at ??:0
@ 0x7f3a249e2061 kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
@ 0x7f3a249e1e6e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f3a1fedf8cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f3a1fedf4ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f3a1fecb4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f3a1fecba2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f3a1fe6851a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f3a1fe66f45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240503 02:12:19.506726 25944 leader_election.cc:302] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221; no voters: 2cf1cac003b44643a2d4de1204aacf0f, ae8b30bcfa5b425782a95374787dd230, cff7b7a19fc64db5849391b1d8e10cce
I20240503 02:12:19.506992 28436 raft_consensus.cc:2726] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:19.507021 26003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e7ddb56be1499b94920ebd252595ce" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 2 candidate_status { last_received { term: 1 index: 170 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:12:19.507615 28673 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 232, Time since last communication: 0.000s
I20240503 02:12:19.508206 28673 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 232, Time since last communication: 0.000s
I20240503 02:12:19.511379 26334 leader_election.cc:302] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, cff7b7a19fc64db5849391b1d8e10cce; no voters: 2cf1cac003b44643a2d4de1204aacf0f, 7c384005cb3041a0bd70e163473b6acb, ae8b30bcfa5b425782a95374787dd230
I20240503 02:12:19.511850 28410 raft_consensus.cc:2726] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:19.519898 28667 leader_election.cc:288] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 2 election: Requested vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:12:19.520957 26075 leader_election.cc:302] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 2cf1cac003b44643a2d4de1204aacf0f; no voters: 3bb103b2fd2344e7b55caf7389acf221, ae8b30bcfa5b425782a95374787dd230, cff7b7a19fc64db5849391b1d8e10cce
I20240503 02:12:19.521693 26393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce"
I20240503 02:12:19.524714 28609 raft_consensus.cc:2726] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:19.527894 26656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
I20240503 02:12:19.534685 28624 ts_tablet_manager.cc:1398] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20240503 02:12:19.534969 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e7ddb56be1499b94920ebd252595ce" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 2 candidate_status { last_received { term: 1 index: 170 } } ignore_live_leader: false dest_uuid: "0a801f0897884f108cdb89434b18628a" is_pre_election: true
I20240503 02:12:19.535908 26159 tablet_copy_service.cc:342] P 2cf1cac003b44643a2d4de1204aacf0f: Request end of tablet copy session 0a801f0897884f108cdb89434b18628a-c12cc2e0fb09431e95baae55803f2184 received from {username='slave'} at 127.23.104.199:43229
I20240503 02:12:19.536041 26159 tablet_copy_service.cc:433] P 2cf1cac003b44643a2d4de1204aacf0f: ending tablet copy session 0a801f0897884f108cdb89434b18628a-c12cc2e0fb09431e95baae55803f2184 on tablet c12cc2e0fb09431e95baae55803f2184 with peer 0a801f0897884f108cdb89434b18628a
I20240503 02:12:19.536320 26592 leader_election.cc:302] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ae8b30bcfa5b425782a95374787dd230; no voters: 0a801f0897884f108cdb89434b18628a, 3bb103b2fd2344e7b55caf7389acf221, b2835e59d40a48aea53c5dedb3bd1e9e
I20240503 02:12:19.593482 28673 raft_consensus.cc:2726] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:19.603099 28664 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35910 (ReqId={client: 6d4bb856b5db4024baf5e69853284c81, seq_no=739, attempt_no=0}) took 2264 ms. Trace:
I20240503 02:12:19.602676 28673 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 1 LEADER]: Committing config change with OpId 1.234: config changed from index -1 to 234, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:19.603209 28664 rpcz_store.cc:276] 0503 02:12:17.338701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:17.338749 (+ 48us) service_pool.cc:225] Handling call
0503 02:12:19.603047 (+2264298us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 02:12:17.339343 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 4a864ac954e046eda8925f5daf4a0f1f
0503 02:12:17.339456 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0503 02:12:17.339461 (+ 5us) write_op.cc:435] Acquired schema lock
0503 02:12:17.339464 (+ 3us) tablet.cc:660] Decoding operations
0503 02:12:17.339529 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0503 02:12:17.339540 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0503 02:12:17.339544 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0503 02:12:17.339580 (+ 36us) tablet.cc:699] Row locks acquired
0503 02:12:17.339582 (+ 2us) write_op.cc:260] PREPARE: finished
0503 02:12:17.339614 (+ 32us) write_op.cc:270] Start()
0503 02:12:17.339635 (+ 21us) write_op.cc:276] Timestamp: P: 1714702337339584 usec, L: 0
0503 02:12:17.339638 (+ 3us) op_driver.cc:331] REPLICATION: starting
0503 02:12:17.339800 (+ 162us) log.cc:842] Serialized 357 byte log entry
0503 02:12:19.602533 (+2262733us) op_driver.cc:444] REPLICATION: finished
0503 02:12:19.602624 (+ 91us) write_op.cc:301] APPLY: starting
0503 02:12:19.602638 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0503 02:12:19.602671 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0503 02:12:19.602699 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0503 02:12:19.602747 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
0503 02:12:19.602773 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0503 02:12:19.602780 (+ 7us) write_op.cc:312] APPLY: finished
0503 02:12:19.602838 (+ 58us) log.cc:842] Serialized 23 byte log entry
0503 02:12:19.602951 (+ 113us) write_op.cc:489] Releasing partition, row and schema locks
0503 02:12:19.602970 (+ 19us) write_op.cc:454] Released schema lock
0503 02:12:19.602976 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":48,"num_ops":1,"prepare.queue_time_us":189,"prepare.run_cpu_time_us":555,"prepare.run_wall_time_us":555,"raft.queue_time_us":2046,"raft.run_cpu_time_us":668,"raft.run_wall_time_us":668,"replication_time_us":2262876}]]}
I20240503 02:12:19.606535 28673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:48140 (request call id 1) took 2266 ms. Trace:
I20240503 02:12:19.606653 28673 rpcz_store.cc:276] 0503 02:12:17.339623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:17.340277 (+ 654us) service_pool.cc:225] Handling call
0503 02:12:18.872329 (+1532052us) log.cc:842] Serialized 755 byte log entry
0503 02:12:19.606484 (+734155us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":311,"raft.queue_time_us":205,"raft.run_cpu_time_us":851,"raft.run_wall_time_us":898,"spinlock_wait_cycles":339200,"thread_start_us":27950,"threads_started":1,"wal-append.queue_time_us":27525}
I20240503 02:12:19.627741 26397 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index -1 to 234, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:19.637583 25859 catalog_manager.cc:5484] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: config changed from index -1 to 234, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "ae8b30bcfa5b425782a95374787dd230" committed_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:19.638806 26139 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index -1 to 234, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
W20240503 02:12:19.651901 26202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 151139us
I20240503 02:12:19.666330 26004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221"
I20240503 02:12:19.669517 28576 consensus_queue.cc:1046] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: INVALID_TERM, Last received: 0.0, Next index: 234, Last known committed idx: 232, Time since last communication: 0.000s
I20240503 02:12:19.670014 26269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e7ddb56be1499b94920ebd252595ce" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 2 candidate_status { last_received { term: 1 index: 170 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:12:19.672689 26202 leader_election.cc:302] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 7c384005cb3041a0bd70e163473b6acb; no voters: 3bb103b2fd2344e7b55caf7389acf221, ae8b30bcfa5b425782a95374787dd230, cff7b7a19fc64db5849391b1d8e10cce
I20240503 02:12:19.674538 28673 raft_consensus.cc:3032] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:19.674659 28673 raft_consensus.cc:729] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Running, Role: LEADER
I20240503 02:12:19.674950 28673 consensus_queue.cc:260] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 1.234, Last appended by leader: 234, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:19.675518 28673 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:19.687108 28667 raft_consensus.cc:2726] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240503 02:12:19.720536 26619 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35910: Illegal state: replica ae8b30bcfa5b425782a95374787dd230 is not leader of this config: current role FOLLOWER
W20240503 02:12:19.734019 26359 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37046: Illegal state: replica cff7b7a19fc64db5849391b1d8e10cce is not leader of this config: current role FOLLOWER
W20240503 02:12:19.783537 26074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 134352us
I20240503 02:12:19.787147 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 2 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f"
W20240503 02:12:19.808912 26202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 92386.3us
I20240503 02:12:19.841096 26659 consensus_queue.cc:237] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 1.282, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:19.845011 26525 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20240503 02:12:19.845552 28673 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20240503 02:12:19.847324 26397 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20240503 02:12:19.847913 28673 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20240503 02:12:19.851939 26003 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20240503 02:12:19.852582 28673 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20240503 02:12:19.853686 28722 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 1 LEADER]: Committing config change with OpId 1.283: config changed from index -1 to 283, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:19.856845 26525 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index -1 to 283, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:19.864796 26399 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index -1 to 283, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:19.868569 26139 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20240503 02:12:19.874369 25861 catalog_manager.cc:5484] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e reported cstate change: config changed from index -1 to 283, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "ae8b30bcfa5b425782a95374787dd230" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:19.896412 28722 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20240503 02:12:19.897037 26005 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index -1 to 283, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
W20240503 02:12:19.896237 26592 consensus_peers.cc:485] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7eb1a7e7a1a440c1a826ac2f2e27db2d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:19.902153 26133 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index -1 to 283, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:19.931388 27591 raft_consensus.cc:1203] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.126->[1.127-1.136] Dedup: 1.132->[1.133-1.136]
W20240503 02:12:20.006255 26229 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49488: Illegal state: replica 7c384005cb3041a0bd70e163473b6acb is not leader of this config: current role FOLLOWER
W20240503 02:12:20.083657 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 78056.4us
W20240503 02:12:20.108068 25968 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53064: Illegal state: replica 3bb103b2fd2344e7b55caf7389acf221 is not leader of this config: current role FOLLOWER
I20240503 02:12:20.416697 28532 raft_consensus.cc:1050] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f: attempting to promote NON_VOTER 0a801f0897884f108cdb89434b18628a to VOTER
I20240503 02:12:20.419626 28532 consensus_queue.cc:237] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 138, Committed index: 138, Last appended: 1.138, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:20.424028 26264 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240503 02:12:20.424428 26525 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240503 02:12:20.424537 28609 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 138, Time since last communication: 0.000s
I20240503 02:12:20.426764 26391 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240503 02:12:20.427320 28609 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 137, Time since last communication: 0.000s
I20240503 02:12:20.429208 28609 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 138, Time since last communication: 0.000s
I20240503 02:12:20.435101 28658 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Committing config change with OpId 1.139: config changed from index 127 to 139, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:20.440105 27590 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240503 02:12:20.440706 26005 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240503 02:12:20.443140 26399 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index 127 to 139, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:20.453334 25858 catalog_manager.cc:5484] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: config changed from index 127 to 139, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:20.490078 28609 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 137, Time since last communication: 0.000s
I20240503 02:12:20.490398 28619 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 137, Time since last communication: 0.000s
I20240503 02:12:20.506057 26265 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index 127 to 139, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:20.510437 27594 raft_consensus.cc:1203] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.137->[1.138-1.139] Dedup: 1.138->[1.139-1.139]
I20240503 02:12:20.511629 27594 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index 127 to 139, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:20.512209 26523 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index 127 to 139, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:20.531169 26099 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57060: Illegal state: replica 2cf1cac003b44643a2d4de1204aacf0f is not leader of this config: current role FOLLOWER
W20240503 02:12:20.547387 26619 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35910: Illegal state: replica ae8b30bcfa5b425782a95374787dd230 is not leader of this config: current role FOLLOWER
W20240503 02:12:20.558408 26359 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37046: Illegal state: replica cff7b7a19fc64db5849391b1d8e10cce is not leader of this config: current role FOLLOWER
W20240503 02:12:20.570677 26229 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49488: Illegal state: replica 7c384005cb3041a0bd70e163473b6acb is not leader of this config: current role FOLLOWER
I20240503 02:12:20.574659 26003 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index 127 to 139, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:20.581784 25968 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53064: Illegal state: replica 3bb103b2fd2344e7b55caf7389acf221 is not leader of this config: current role FOLLOWER
W20240503 02:12:20.607599 26099 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57060: Illegal state: replica 2cf1cac003b44643a2d4de1204aacf0f is not leader of this config: current role FOLLOWER
W20240503 02:12:20.625365 26619 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35910: Illegal state: replica ae8b30bcfa5b425782a95374787dd230 is not leader of this config: current role FOLLOWER
W20240503 02:12:20.640192 26359 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37046: Illegal state: replica cff7b7a19fc64db5849391b1d8e10cce is not leader of this config: current role FOLLOWER
I20240503 02:12:20.646886 28785 ts_tablet_manager.cc:902] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a: Initiating tablet copy from peer ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369)
W20240503 02:12:20.657146 26228 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49488: Illegal state: replica 7c384005cb3041a0bd70e163473b6acb is not leader of this config: current role FOLLOWER
I20240503 02:12:20.666154 26397 raft_consensus.cc:1203] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.137->[1.138-1.138] Dedup: 1.138->[]
W20240503 02:12:20.676247 25968 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53064: Illegal state: replica 3bb103b2fd2344e7b55caf7389acf221 is not leader of this config: current role FOLLOWER
I20240503 02:12:20.722927 28785 tablet_copy_client.cc:308] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:40369
W20240503 02:12:20.736956 26097 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57060: Illegal state: replica 2cf1cac003b44643a2d4de1204aacf0f is not leader of this config: current role FOLLOWER
I20240503 02:12:20.755592 26679 tablet_copy_service.cc:140] P ae8b30bcfa5b425782a95374787dd230: Received BeginTabletCopySession request for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d from peer 0a801f0897884f108cdb89434b18628a ({username='slave'} at 127.23.104.199:41791)
I20240503 02:12:20.755816 26679 tablet_copy_service.cc:161] P ae8b30bcfa5b425782a95374787dd230: Beginning new tablet copy session on tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d from peer 0a801f0897884f108cdb89434b18628a at {username='slave'} at 127.23.104.199:41791: session id = 0a801f0897884f108cdb89434b18628a-7eb1a7e7a1a440c1a826ac2f2e27db2d
I20240503 02:12:20.760003 26679 tablet_copy_source_session.cc:205] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:20.761950 28785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.767603 28785 tablet_copy_client.cc:783] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:20.767984 28785 tablet_copy_client.cc:647] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:20.771332 28785 tablet_copy_client.cc:523] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 02:12:20.774322 26619 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35910: Illegal state: replica ae8b30bcfa5b425782a95374787dd230 is not leader of this config: current role FOLLOWER
I20240503 02:12:20.776177 28785 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a: Bootstrap starting.
W20240503 02:12:20.792764 26359 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37046: Illegal state: replica cff7b7a19fc64db5849391b1d8e10cce is not leader of this config: current role FOLLOWER
W20240503 02:12:20.812558 26228 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49488: Illegal state: replica 7c384005cb3041a0bd70e163473b6acb is not leader of this config: current role FOLLOWER
I20240503 02:12:20.837081 26133 consensus_queue.cc:237] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 142, Committed index: 142, Last appended: 1.142, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
W20240503 02:12:20.839129 25968 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53064: Illegal state: replica 3bb103b2fd2344e7b55caf7389acf221 is not leader of this config: current role FOLLOWER
I20240503 02:12:20.867887 27594 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 1 index: 143. (index mismatch)
I20240503 02:12:20.867535 26004 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 141. Preceding OpId from leader: term: 1 index: 143. (index mismatch)
I20240503 02:12:21.017735 28436 raft_consensus.cc:482] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae8b30bcfa5b425782a95374787dd230)
I20240503 02:12:21.501722 28436 raft_consensus.cc:504] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:21.502691 28436 leader_election.cc:288] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:12:21.555863 26264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:12:21.556156 26264 raft_consensus.cc:2379] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3bb103b2fd2344e7b55caf7389acf221 in current term 2: Already voted for candidate 7c384005cb3041a0bd70e163473b6acb in this term.
I20240503 02:12:21.611586 28667 raft_consensus.cc:482] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:21.611881 28667 raft_consensus.cc:504] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } }
I20240503 02:12:21.621364 26003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 3 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:12:21.685175 26003 raft_consensus.cc:2396] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c384005cb3041a0bd70e163473b6acb for term 3 because replica has last-logged OpId of term: 1 index: 234, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 233.
I20240503 02:12:21.620260 28667 leader_election.cc:288] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:12:20.860067 26525 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 1 index: 143. (index mismatch)
I20240503 02:12:21.364900 26528 consensus_queue.cc:237] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 1.250, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:20.860805 26397 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 1 index: 143. (index mismatch)
I20240503 02:12:21.128397 28625 raft_consensus.cc:482] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae8b30bcfa5b425782a95374787dd230)
I20240503 02:12:21.685873 28625 raft_consensus.cc:504] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:21.390972 28722 raft_consensus.cc:482] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:21.686628 28722 raft_consensus.cc:504] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:21.687245 28625 leader_election.cc:288] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:12:21.687605 28722 leader_election.cc:288] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
W20240503 02:12:20.868794 26097 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57060: Illegal state: replica 2cf1cac003b44643a2d4de1204aacf0f is not leader of this config: current role FOLLOWER
I20240503 02:12:21.503264 26657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:12:21.688052 26003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:12:21.688206 26657 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb103b2fd2344e7b55caf7389acf221 in term 2.
I20240503 02:12:21.688251 26003 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 1.
I20240503 02:12:21.641845 26656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 3 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:12:21.689021 26656 raft_consensus.cc:2396] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c384005cb3041a0bd70e163473b6acb for term 3 because replica has last-logged OpId of term: 1 index: 234, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 233.
I20240503 02:12:21.689074 26397 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 252. (index mismatch)
I20240503 02:12:21.148252 28786 raft_consensus.cc:482] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae8b30bcfa5b425782a95374787dd230)
I20240503 02:12:21.689513 28786 raft_consensus.cc:504] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:21.689677 26657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:12:21.689852 26657 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 2.
I20240503 02:12:21.689947 26003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 3 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:12:21.690121 26003 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 1.
I20240503 02:12:21.690546 28786 leader_election.cc:288] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143)
I20240503 02:12:21.690533 26397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 3 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:12:21.690721 26397 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 1.
I20240503 02:12:21.691196 26269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 3 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:12:21.691426 26269 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 2.
I20240503 02:12:21.691426 26332 leader_election.cc:302] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, ae8b30bcfa5b425782a95374787dd230, cff7b7a19fc64db5849391b1d8e10cce; no voters:
I20240503 02:12:21.691498 28560 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 249, Time since last communication: 0.000s
I20240503 02:12:21.691661 28625 raft_consensus.cc:2781] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:12:21.691718 26265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:12:21.691772 28625 raft_consensus.cc:482] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae8b30bcfa5b425782a95374787dd230)
I20240503 02:12:21.691888 28625 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:21.691979 26265 raft_consensus.cc:2379] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2cf1cac003b44643a2d4de1204aacf0f in current term 2: Already voted for candidate 7c384005cb3041a0bd70e163473b6acb in this term.
I20240503 02:12:21.692341 26264 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 252. (index mismatch)
I20240503 02:12:21.692716 26592 leader_election.cc:302] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb, ae8b30bcfa5b425782a95374787dd230; no voters:
I20240503 02:12:21.692860 28669 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 249, Time since last communication: 0.000s
I20240503 02:12:21.693116 26657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:12:21.693292 26657 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf1cac003b44643a2d4de1204aacf0f in term 2.
I20240503 02:12:21.693878 26656 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 252. (index mismatch)
I20240503 02:12:21.694401 28722 raft_consensus.cc:2781] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:12:21.694505 28722 raft_consensus.cc:482] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:21.694584 28722 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:21.695161 28625 raft_consensus.cc:504] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 2 FOLLOWER]: Starting leader election with config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:21.695631 28669 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240503 02:12:21.696372 28625 leader_election.cc:288] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 2 election: Requested vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:12:21.696667 26003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221"
I20240503 02:12:21.696858 26003 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:21.697167 28722 raft_consensus.cc:504] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:21.698299 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 3 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce"
I20240503 02:12:21.698493 26391 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:21.698807 26656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
I20240503 02:12:21.698990 26656 raft_consensus.cc:2354] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cff7b7a19fc64db5849391b1d8e10cce for earlier term 2. Current term is 3.
I20240503 02:12:21.699620 26335 leader_election.cc:398] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 2 election: Vote denied by peer ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369) with higher term. Message: Invalid argument: T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cff7b7a19fc64db5849391b1d8e10cce for earlier term 2. Current term is 3.
I20240503 02:12:21.699733 26335 leader_election.cc:401] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240503 02:12:21.700785 26265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 3 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb"
I20240503 02:12:21.701033 26265 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:21.702024 26391 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 3.
I20240503 02:12:21.702366 26004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 3 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221"
I20240503 02:12:21.704022 26265 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 3.
I20240503 02:12:21.704432 26594 leader_election.cc:302] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 7c384005cb3041a0bd70e163473b6acb, ae8b30bcfa5b425782a95374787dd230, cff7b7a19fc64db5849391b1d8e10cce; no voters: 3bb103b2fd2344e7b55caf7389acf221
I20240503 02:12:21.704627 28625 raft_consensus.cc:2726] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369) with higher term. Message: Invalid argument: T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cff7b7a19fc64db5849391b1d8e10cce for earlier term 2. Current term is 3.
I20240503 02:12:21.704777 28673 raft_consensus.cc:2781] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:12:21.704893 28673 raft_consensus.cc:686] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 LEADER]: Becoming Leader. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Running, Role: LEADER
I20240503 02:12:21.705291 28673 consensus_queue.cc:237] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 1.234, Last appended by leader: 234, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:21.705574 28669 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 LEADER]: Committing config change with OpId 1.251: config changed from index -1 to 251, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:21.705962 26264 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index -1 to 251, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:21.708762 26656 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index -1 to 251, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:21.710007 26003 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff7b7a19fc64db5849391b1d8e10cce in term 2.
I20240503 02:12:21.711341 28722 leader_election.cc:288] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 3 election: Requested vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:12:21.711869 26392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:12:21.712109 26392 raft_consensus.cc:2354] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2cf1cac003b44643a2d4de1204aacf0f for earlier term 2. Current term is 3.
I20240503 02:12:21.712807 26397 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index -1 to 251, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:21.504379 26393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:12:21.624605 26395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 3 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce" is_pre_election: true
I20240503 02:12:21.717311 26395 raft_consensus.cc:2379] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c384005cb3041a0bd70e163473b6acb in current term 3: Already voted for candidate ae8b30bcfa5b425782a95374787dd230 in this term.
I20240503 02:12:21.718016 26204 leader_election.cc:302] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 7c384005cb3041a0bd70e163473b6acb; no voters: 3bb103b2fd2344e7b55caf7389acf221, ae8b30bcfa5b425782a95374787dd230, cff7b7a19fc64db5849391b1d8e10cce
I20240503 02:12:21.717819 25858 catalog_manager.cc:5484] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ae8b30bcfa5b425782a95374787dd230" committed_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:21.718479 28787 raft_consensus.cc:2726] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:12:21.718438 26269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb"
I20240503 02:12:21.718703 26269 raft_consensus.cc:2354] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cff7b7a19fc64db5849391b1d8e10cce for earlier term 2. Current term is 3.
I20240503 02:12:21.718772 26393 raft_consensus.cc:2354] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3bb103b2fd2344e7b55caf7389acf221 for earlier term 2. Current term is 3.
I20240503 02:12:21.719089 26334 leader_election.cc:398] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 2 election: Vote denied by peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999) with higher term. Message: Invalid argument: T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cff7b7a19fc64db5849391b1d8e10cce for earlier term 2. Current term is 3.
I20240503 02:12:21.719640 26074 leader_election.cc:398] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 2 pre-election: Vote denied by peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675) with higher term. Message: Invalid argument: T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2cf1cac003b44643a2d4de1204aacf0f for earlier term 2. Current term is 3.
I20240503 02:12:21.719643 25943 leader_election.cc:398] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 2 pre-election: Vote denied by peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675) with higher term. Message: Invalid argument: T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3bb103b2fd2344e7b55caf7389acf221 for earlier term 2. Current term is 3.
I20240503 02:12:21.719729 26074 leader_election.cc:401] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240503 02:12:21.719729 25943 leader_election.cc:401] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240503 02:12:21.720028 28436 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:21.720204 28619 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Advancing to term 3
I20240503 02:12:21.722843 28619 raft_consensus.cc:2726] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675) with higher term. Message: Invalid argument: T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2cf1cac003b44643a2d4de1204aacf0f for earlier term 2. Current term is 3.
I20240503 02:12:21.724481 26265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "7c384005cb3041a0bd70e163473b6acb" is_pre_election: true
I20240503 02:12:21.724675 26265 raft_consensus.cc:2354] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cff7b7a19fc64db5849391b1d8e10cce for earlier term 2. Current term is 3.
I20240503 02:12:21.725191 28436 raft_consensus.cc:2726] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675) with higher term. Message: Invalid argument: T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3bb103b2fd2344e7b55caf7389acf221 for earlier term 2. Current term is 3.
I20240503 02:12:21.726495 26334 leader_election.cc:398] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999) with higher term. Message: Invalid argument: T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cff7b7a19fc64db5849391b1d8e10cce for earlier term 2. Current term is 3.
W20240503 02:12:21.539498 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 58280.9us
I20240503 02:12:21.620913 26137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "7c384005cb3041a0bd70e163473b6acb" candidate_term: 3 candidate_status { last_received { term: 1 index: 233 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:12:21.736059 26137 raft_consensus.cc:2396] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c384005cb3041a0bd70e163473b6acb for term 3 because replica has last-logged OpId of term: 1 index: 234, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 233.
I20240503 02:12:21.686553 28532 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 142, Time since last communication: 0.000s
I20240503 02:12:21.736909 28537 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 142, Time since last communication: 0.000s
I20240503 02:12:21.737843 28532 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 141, Time since last communication: 0.000s
I20240503 02:12:21.743961 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:12:21.744180 26133 raft_consensus.cc:2354] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cff7b7a19fc64db5849391b1d8e10cce for earlier term 2. Current term is 3.
I20240503 02:12:21.744813 26137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "cff7b7a19fc64db5849391b1d8e10cce" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f"
I20240503 02:12:21.745019 26137 raft_consensus.cc:2354] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cff7b7a19fc64db5849391b1d8e10cce for earlier term 2. Current term is 3.
I20240503 02:12:21.748023 26332 leader_election.cc:398] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 2 pre-election: Vote denied by peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753) with higher term. Message: Invalid argument: T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cff7b7a19fc64db5849391b1d8e10cce for earlier term 2. Current term is 3.
I20240503 02:12:21.751140 28532 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Committing config change with OpId 1.143: config changed from index 139 to 143, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:21.754691 26137 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 252. (index mismatch)
I20240503 02:12:21.757133 25863 catalog_manager.cc:5484] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: config changed from index -1 to 251, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:21.774874 26003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "2cf1cac003b44643a2d4de1204aacf0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "3bb103b2fd2344e7b55caf7389acf221" is_pre_election: true
I20240503 02:12:21.776106 25851 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c12cc2e0fb09431e95baae55803f2184 with cas_config_opid_index 139: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:21.776208 26003 raft_consensus.cc:2354] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2cf1cac003b44643a2d4de1204aacf0f for earlier term 2. Current term is 3.
I20240503 02:12:21.777448 26005 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 3 index: 236. (index mismatch)
I20240503 02:12:21.777758 26265 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 3 index: 236. (index mismatch)
I20240503 02:12:21.778232 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 3 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:12:21.778515 26133 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 3.
I20240503 02:12:21.778687 28717 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 232, Time since last communication: 0.000s
W20240503 02:12:21.779668 26592 consensus_peers.cc:485] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a864ac954e046eda8925f5daf4a0f1f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:21.779915 28717 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 232, Time since last communication: 0.000s
I20240503 02:12:21.781360 26005 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 3 FOLLOWER]: Committing config change with OpId 1.234: config changed from index -1 to 234, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:21.784070 26528 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.143: config changed from index 139 to 143, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:21.787415 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 3 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f"
I20240503 02:12:21.789217 26397 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.143: config changed from index 139 to 143, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:21.793488 26133 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 3.
I20240503 02:12:21.796798 26133 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 3 index: 236. (index mismatch)
I20240503 02:12:21.808126 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "3bb103b2fd2344e7b55caf7389acf221" candidate_term: 2 candidate_status { last_received { term: 1 index: 234 } } ignore_live_leader: false dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f" is_pre_election: true
I20240503 02:12:21.815018 26265 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Committing config change with OpId 1.234: config changed from index -1 to 234, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:21.818035 26133 raft_consensus.cc:2354] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3bb103b2fd2344e7b55caf7389acf221 for earlier term 2. Current term is 3.
I20240503 02:12:21.836145 28619 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 141, Time since last communication: 0.000s
I20240503 02:12:21.861552 26332 leader_election.cc:398] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [CANDIDATE]: Term 2 election: Vote denied by peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753) with higher term. Message: Invalid argument: T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cff7b7a19fc64db5849391b1d8e10cce for earlier term 2. Current term is 3.
I20240503 02:12:21.862680 28669 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240503 02:12:21.902381 28722 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240503 02:12:21.904608 25941 leader_election.cc:398] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753) with higher term. Message: Invalid argument: T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3bb103b2fd2344e7b55caf7389acf221 for earlier term 2. Current term is 3.
I20240503 02:12:21.915891 25862 catalog_manager.cc:5484] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f reported cstate change: config changed from index 139 to 143, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:21.925858 26072 leader_election.cc:398] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143) with higher term. Message: Invalid argument: T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2cf1cac003b44643a2d4de1204aacf0f for earlier term 2. Current term is 3.
I20240503 02:12:21.936475 26007 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.143: config changed from index 139 to 143, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:22.030807 26265 consensus_queue.cc:237] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 156, Committed index: 156, Last appended: 1.156, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:22.032775 26391 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 3 index: 236. (index mismatch)
I20240503 02:12:22.033252 26136 raft_consensus.cc:1261] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 02:12:22.033569 28717 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240503 02:12:22.035121 28787 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240503 02:12:22.036644 26525 raft_consensus.cc:1261] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 02:12:22.037158 28787 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240503 02:12:22.040838 26657 raft_consensus.cc:1261] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 02:12:22.041570 28867 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Committing config change with OpId 1.157: config changed from index -1 to 157, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:22.044667 26136 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index -1 to 251, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:22.045615 28667 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240503 02:12:22.047250 26657 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index -1 to 157, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
W20240503 02:12:22.054316 26202 consensus_peers.cc:485] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 740218c49aa84ee28f875f2cfcee01b8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:22.054500 26525 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index -1 to 157, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:22.063321 26137 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index -1 to 157, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:22.107096 26007 raft_consensus.cc:1261] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 02:12:22.109808 28770 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240503 02:12:22.112646 26008 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index -1 to 157, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
W20240503 02:12:22.155740 26462 consensus_peers.cc:485] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 508ee4b913dd40b49d50be3fa4cdd6c4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:22.199050 27591 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.143: config changed from index 139 to 143, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:22.267072 25858 catalog_manager.cc:5484] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230 reported cstate change: config changed from index -1 to 157, NON_VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } } }
I20240503 02:12:22.278035 26247 tablet_service.cc:1514] Processing DeleteTablet for tablet c12cc2e0fb09431e95baae55803f2184 with delete_type TABLET_DATA_TOMBSTONED (TS 7c384005cb3041a0bd70e163473b6acb not found in new config with opid_index 143) from {username='slave'} at 127.0.0.1:37992
I20240503 02:12:22.314203 28928 tablet_replica.cc:330] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb: stopping tablet replica
I20240503 02:12:22.314539 28928 raft_consensus.cc:2229] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:22.314883 28928 raft_consensus.cc:2258] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:22.335095 28929 ts_tablet_manager.cc:902] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a: Initiating tablet copy from peer ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369)
I20240503 02:12:22.372191 28926 ts_tablet_manager.cc:902] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a: Initiating tablet copy from peer b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347)
I20240503 02:12:22.429199 28928 ts_tablet_manager.cc:1875] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:22.459075 28946 ts_tablet_manager.cc:902] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a: Initiating tablet copy from peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999)
I20240503 02:12:22.549557 28929 tablet_copy_client.cc:308] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:40369
I20240503 02:12:22.551086 26679 tablet_copy_service.cc:140] P ae8b30bcfa5b425782a95374787dd230: Received BeginTabletCopySession request for tablet 4a864ac954e046eda8925f5daf4a0f1f from peer 0a801f0897884f108cdb89434b18628a ({username='slave'} at 127.23.104.199:41791)
I20240503 02:12:22.551334 26679 tablet_copy_service.cc:161] P ae8b30bcfa5b425782a95374787dd230: Beginning new tablet copy session on tablet 4a864ac954e046eda8925f5daf4a0f1f from peer 0a801f0897884f108cdb89434b18628a at {username='slave'} at 127.23.104.199:41791: session id = 0a801f0897884f108cdb89434b18628a-4a864ac954e046eda8925f5daf4a0f1f
I20240503 02:12:22.555461 26679 tablet_copy_source_session.cc:205] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:22.556941 28929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a864ac954e046eda8925f5daf4a0f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.577018 28946 tablet_copy_client.cc:308] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:45999
I20240503 02:12:22.577746 26289 tablet_copy_service.cc:140] P 7c384005cb3041a0bd70e163473b6acb: Received BeginTabletCopySession request for tablet 740218c49aa84ee28f875f2cfcee01b8 from peer 0a801f0897884f108cdb89434b18628a ({username='slave'} at 127.23.104.199:38513)
I20240503 02:12:22.577903 26289 tablet_copy_service.cc:161] P 7c384005cb3041a0bd70e163473b6acb: Beginning new tablet copy session on tablet 740218c49aa84ee28f875f2cfcee01b8 from peer 0a801f0897884f108cdb89434b18628a at {username='slave'} at 127.23.104.199:38513: session id = 0a801f0897884f108cdb89434b18628a-740218c49aa84ee28f875f2cfcee01b8
I20240503 02:12:22.581504 26289 tablet_copy_source_session.cc:205] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:22.585286 28929 tablet_copy_client.cc:783] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:22.585551 28929 tablet_copy_client.cc:647] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:22.585813 28946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740218c49aa84ee28f875f2cfcee01b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.588445 28929 tablet_copy_client.cc:523] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:22.588809 28928 ts_tablet_manager.cc:1888] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.142
I20240503 02:12:22.588989 28928 log.cc:1196] T c12cc2e0fb09431e95baae55803f2184 P 7c384005cb3041a0bd70e163473b6acb: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/c12cc2e0fb09431e95baae55803f2184
I20240503 02:12:22.592943 25853 catalog_manager.cc:4830] TS 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999): tablet c12cc2e0fb09431e95baae55803f2184 (table rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]) successfully deleted
I20240503 02:12:22.614450 28946 tablet_copy_client.cc:783] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:22.614708 28946 tablet_copy_client.cc:647] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:22.620716 28929 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a: Bootstrap starting.
I20240503 02:12:22.623301 28946 tablet_copy_client.cc:523] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:22.628670 28946 tablet_bootstrap.cc:492] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a: Bootstrap starting.
W20240503 02:12:22.694572 26202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 77882.4us
W20240503 02:12:22.858312 26592 consensus_peers.cc:485] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 02:12:23.059895 28926 tablet_copy_client.cc:308] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.197:38347
I20240503 02:12:23.157734 26549 tablet_copy_service.cc:140] P b2835e59d40a48aea53c5dedb3bd1e9e: Received BeginTabletCopySession request for tablet 508ee4b913dd40b49d50be3fa4cdd6c4 from peer 0a801f0897884f108cdb89434b18628a ({username='slave'} at 127.23.104.199:35337)
I20240503 02:12:23.157989 26549 tablet_copy_service.cc:161] P b2835e59d40a48aea53c5dedb3bd1e9e: Beginning new tablet copy session on tablet 508ee4b913dd40b49d50be3fa4cdd6c4 from peer 0a801f0897884f108cdb89434b18628a at {username='slave'} at 127.23.104.199:35337: session id = 0a801f0897884f108cdb89434b18628a-508ee4b913dd40b49d50be3fa4cdd6c4
I20240503 02:12:23.162068 26549 tablet_copy_source_session.cc:205] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:23.164862 28926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508ee4b913dd40b49d50be3fa4cdd6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.171854 28926 tablet_copy_client.cc:783] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:23.172067 28926 tablet_copy_client.cc:647] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:23.178905 28926 tablet_copy_client.cc:523] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:23.182356 28926 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a: Bootstrap starting.
W20240503 02:12:23.285609 26202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 49198.4us
W20240503 02:12:23.407502 26202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 62637.2us
W20240503 02:12:23.484439 28982 log.cc:924] Time spent T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
I20240503 02:12:23.704927 28785 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a: Bootstrap replayed 1/1 log segments. Stats: ops{read=283 overwritten=0 applied=283 ignored=0} inserts{seen=955 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:23.705410 28785 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a: Bootstrap complete.
I20240503 02:12:23.705760 28785 ts_tablet_manager.cc:1367] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a: Time spent bootstrapping tablet: real 2.930s user 0.350s sys 0.028s
I20240503 02:12:23.706946 28785 raft_consensus.cc:348] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:23.707254 28785 raft_consensus.cc:729] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0a801f0897884f108cdb89434b18628a, State: Initialized, Role: LEARNER
I20240503 02:12:23.707473 28785 consensus_queue.cc:260] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 283, Last appended: 1.283, Last appended by leader: 283, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:23.718874 28785 ts_tablet_manager.cc:1398] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a: Time spent starting tablet: real 0.013s user 0.002s sys 0.002s
I20240503 02:12:23.724691 26679 tablet_copy_service.cc:342] P ae8b30bcfa5b425782a95374787dd230: Request end of tablet copy session 0a801f0897884f108cdb89434b18628a-7eb1a7e7a1a440c1a826ac2f2e27db2d received from {username='slave'} at 127.23.104.199:41791
I20240503 02:12:23.725054 26679 tablet_copy_service.cc:433] P ae8b30bcfa5b425782a95374787dd230: ending tablet copy session 0a801f0897884f108cdb89434b18628a-7eb1a7e7a1a440c1a826ac2f2e27db2d on tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d with peer 0a801f0897884f108cdb89434b18628a
W20240503 02:12:24.016268 26202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 78204.8us
I20240503 02:12:24.023880 28946 tablet_bootstrap.cc:492] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a: Bootstrap replayed 1/1 log segments. Stats: ops{read=161 overwritten=0 applied=161 ignored=0} inserts{seen=792 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:24.024616 28946 tablet_bootstrap.cc:492] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a: Bootstrap complete.
I20240503 02:12:24.024947 28946 ts_tablet_manager.cc:1367] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a: Time spent bootstrapping tablet: real 1.396s user 0.209s sys 0.037s
I20240503 02:12:24.025808 28946 raft_consensus.cc:348] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:24.026103 28946 raft_consensus.cc:729] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0a801f0897884f108cdb89434b18628a, State: Initialized, Role: LEARNER
I20240503 02:12:24.026340 28946 consensus_queue.cc:260] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 161, Last appended: 1.161, Last appended by leader: 161, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:24.043396 28946 ts_tablet_manager.cc:1398] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240503 02:12:24.046593 26289 tablet_copy_service.cc:342] P 7c384005cb3041a0bd70e163473b6acb: Request end of tablet copy session 0a801f0897884f108cdb89434b18628a-740218c49aa84ee28f875f2cfcee01b8 received from {username='slave'} at 127.23.104.199:38513
I20240503 02:12:24.046749 26289 tablet_copy_service.cc:433] P 7c384005cb3041a0bd70e163473b6acb: ending tablet copy session 0a801f0897884f108cdb89434b18628a-740218c49aa84ee28f875f2cfcee01b8 on tablet 740218c49aa84ee28f875f2cfcee01b8 with peer 0a801f0897884f108cdb89434b18628a
I20240503 02:12:24.113041 28929 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a: Bootstrap replayed 1/1 log segments. Stats: ops{read=237 overwritten=0 applied=237 ignored=0} inserts{seen=867 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:24.114156 28929 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a: Bootstrap complete.
I20240503 02:12:24.114593 28929 ts_tablet_manager.cc:1367] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a: Time spent bootstrapping tablet: real 1.494s user 0.302s sys 0.036s
I20240503 02:12:24.116995 28929 raft_consensus.cc:348] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:24.118041 28929 raft_consensus.cc:729] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 0a801f0897884f108cdb89434b18628a, State: Initialized, Role: LEARNER
I20240503 02:12:24.120003 28929 consensus_queue.cc:260] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 237, Last appended: 3.237, Last appended by leader: 237, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:24.124310 28929 ts_tablet_manager.cc:1398] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:12:24.128366 26679 tablet_copy_service.cc:342] P ae8b30bcfa5b425782a95374787dd230: Request end of tablet copy session 0a801f0897884f108cdb89434b18628a-4a864ac954e046eda8925f5daf4a0f1f received from {username='slave'} at 127.23.104.199:41791
I20240503 02:12:24.128481 26679 tablet_copy_service.cc:433] P ae8b30bcfa5b425782a95374787dd230: ending tablet copy session 0a801f0897884f108cdb89434b18628a-4a864ac954e046eda8925f5daf4a0f1f on tablet 4a864ac954e046eda8925f5daf4a0f1f with peer 0a801f0897884f108cdb89434b18628a
I20240503 02:12:24.151185 27591 raft_consensus.cc:1203] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.156->[1.157-1.167] Dedup: 1.161->[1.162-1.167]
I20240503 02:12:24.273509 27592 raft_consensus.cc:1203] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 3 LEARNER]: Deduplicated request from leader. Original: 1.234->[3.235-3.250] Dedup: 3.237->[3.238-3.250]
I20240503 02:12:24.280973 27593 raft_consensus.cc:1203] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.282->[1.283-1.302] Dedup: 1.283->[1.284-1.302]
I20240503 02:12:24.315840 28926 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a: Bootstrap replayed 1/1 log segments. Stats: ops{read=271 overwritten=0 applied=270 ignored=0} inserts{seen=955 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:12:24.316699 28926 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a: Bootstrap complete.
I20240503 02:12:24.317176 28926 ts_tablet_manager.cc:1367] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a: Time spent bootstrapping tablet: real 1.135s user 0.339s sys 0.024s
I20240503 02:12:24.319813 28926 raft_consensus.cc:348] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:24.364600 28970 raft_consensus.cc:1050] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230: attempting to promote NON_VOTER 0a801f0897884f108cdb89434b18628a to VOTER
I20240503 02:12:24.366163 28970 consensus_queue.cc:237] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 3.250, Last appended by leader: 234, Current term: 3, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:24.370566 26265 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 250. Preceding OpId from leader: term: 3 index: 251. (index mismatch)
I20240503 02:12:24.371027 26007 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 250. Preceding OpId from leader: term: 3 index: 251. (index mismatch)
I20240503 02:12:24.372311 26397 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 250. Preceding OpId from leader: term: 3 index: 251. (index mismatch)
I20240503 02:12:24.373200 27592 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 3 LEARNER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 250. Preceding OpId from leader: term: 3 index: 251. (index mismatch)
I20240503 02:12:24.373395 28877 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240503 02:12:24.374683 28877 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240503 02:12:24.375248 28877 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240503 02:12:24.380026 28970 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240503 02:12:24.392331 28722 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 LEADER]: Committing config change with OpId 3.251: config changed from index 234 to 251, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.400427 26269 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Committing config change with OpId 3.251: config changed from index 234 to 251, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.414021 26007 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 3 FOLLOWER]: Committing config change with OpId 3.251: config changed from index 234 to 251, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.423614 27593 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 3 FOLLOWER]: Committing config change with OpId 3.251: config changed from index 234 to 251, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.425628 28969 raft_consensus.cc:1050] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230: attempting to promote NON_VOTER 0a801f0897884f108cdb89434b18628a to VOTER
I20240503 02:12:24.426589 28969 consensus_queue.cc:237] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 1.304, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:24.431563 26391 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Committing config change with OpId 3.251: config changed from index 234 to 251, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.435448 26007 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 304. Preceding OpId from leader: term: 1 index: 305. (index mismatch)
I20240503 02:12:24.436142 25861 catalog_manager.cc:5484] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 reported cstate change: config changed from index 234 to 251, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "ae8b30bcfa5b425782a95374787dd230" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:24.436898 27594 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 305. (index mismatch)
I20240503 02:12:24.440021 28722 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 303, Time since last communication: 0.000s
I20240503 02:12:24.440472 28722 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 303, Time since last communication: 0.000s
I20240503 02:12:24.464892 26392 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 305. (index mismatch)
I20240503 02:12:24.466568 26523 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 304. Preceding OpId from leader: term: 1 index: 305. (index mismatch)
I20240503 02:12:24.466992 28722 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 303, Time since last communication: 0.000s
I20240503 02:12:24.467471 29025 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 303, Time since last communication: 0.000s
I20240503 02:12:24.470268 28970 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 1 LEADER]: Committing config change with OpId 1.305: config changed from index 283 to 305, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.480669 26008 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.305: config changed from index 283 to 305, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.480767 27592 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.305: config changed from index 283 to 305, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.492933 26524 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.305: config changed from index 283 to 305, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.500989 25858 catalog_manager.cc:5484] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e reported cstate change: config changed from index 283 to 305, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae8b30bcfa5b425782a95374787dd230" committed_config { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.524358 26133 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 250. Preceding OpId from leader: term: 3 index: 251. (index mismatch)
I20240503 02:12:24.525422 26133 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 305. (index mismatch)
I20240503 02:12:24.568225 29025 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240503 02:12:24.568742 28722 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 302, Time since last communication: 0.000s
I20240503 02:12:24.579278 28766 raft_consensus.cc:1050] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb: attempting to promote NON_VOTER 0a801f0897884f108cdb89434b18628a to VOTER
I20240503 02:12:24.580636 28766 consensus_queue.cc:237] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 167, Committed index: 167, Last appended: 1.167, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:24.586841 27593 raft_consensus.cc:1261] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 02:12:24.587464 28766 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 02:12:24.587880 26653 raft_consensus.cc:1261] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 02:12:24.588081 26133 raft_consensus.cc:1261] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 02:12:24.588532 28959 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 02:12:24.592583 28667 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 02:12:24.595710 26003 raft_consensus.cc:1261] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 02:12:24.597456 28959 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 02:12:24.599442 28766 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Committing config change with OpId 1.168: config changed from index 157 to 168, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.604347 26524 raft_consensus.cc:1261] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240503 02:12:24.605610 26136 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.305: config changed from index 283 to 305, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.607520 28770 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 02:12:24.609619 26653 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index 157 to 168, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.610162 26397 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.305: config changed from index 283 to 305, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.611579 27593 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index 157 to 168, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.618515 26003 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index 157 to 168, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.625849 26137 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Committing config change with OpId 3.251: config changed from index 234 to 251, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.631093 26133 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index 157 to 168, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.662323 26524 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index 157 to 168, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.670783 28926 raft_consensus.cc:729] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0a801f0897884f108cdb89434b18628a, State: Initialized, Role: LEARNER
I20240503 02:12:24.677776 28926 consensus_queue.cc:260] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 270, Last appended: 1.271, Last appended by leader: 271, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: true } }
I20240503 02:12:24.678464 28926 ts_tablet_manager.cc:1398] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a: Time spent starting tablet: real 0.361s user 0.051s sys 0.081s
I20240503 02:12:24.679844 26549 tablet_copy_service.cc:342] P b2835e59d40a48aea53c5dedb3bd1e9e: Request end of tablet copy session 0a801f0897884f108cdb89434b18628a-508ee4b913dd40b49d50be3fa4cdd6c4 received from {username='slave'} at 127.23.104.199:35337
I20240503 02:12:24.680045 26549 tablet_copy_service.cc:433] P b2835e59d40a48aea53c5dedb3bd1e9e: ending tablet copy session 0a801f0897884f108cdb89434b18628a-508ee4b913dd40b49d50be3fa4cdd6c4 on tablet 508ee4b913dd40b49d50be3fa4cdd6c4 with peer 0a801f0897884f108cdb89434b18628a
I20240503 02:12:24.754757 25863 catalog_manager.cc:5484] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb reported cstate change: config changed from index 157 to 168, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:24.822552 26653 consensus_queue.cc:237] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 1.309, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:24.833567 26525 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 309. Preceding OpId from leader: term: 1 index: 310. (index mismatch)
I20240503 02:12:24.834144 28970 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 310, Last known committed idx: 308, Time since last communication: 0.000s
I20240503 02:12:24.869663 27593 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 309. Preceding OpId from leader: term: 1 index: 310. (index mismatch)
I20240503 02:12:24.869658 26007 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 309. Preceding OpId from leader: term: 1 index: 310. (index mismatch)
I20240503 02:12:24.886767 26136 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 309. Preceding OpId from leader: term: 1 index: 310. (index mismatch)
I20240503 02:12:24.897353 28970 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 310, Last known committed idx: 308, Time since last communication: 0.000s
I20240503 02:12:24.899677 28970 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 310, Last known committed idx: 308, Time since last communication: 0.000s
I20240503 02:12:24.902431 28970 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 310, Last known committed idx: 308, Time since last communication: 0.000s
I20240503 02:12:24.904017 29028 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 1 LEADER]: Committing config change with OpId 1.310: config changed from index 305 to 310, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.910908 25854 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d with cas_config_opid_index 305: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:24.912200 26269 consensus_queue.cc:237] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 174, Committed index: 174, Last appended: 1.175, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:24.915423 26656 raft_consensus.cc:1261] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 174. Preceding OpId from leader: term: 1 index: 176. (index mismatch)
I20240503 02:12:24.915982 28770 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 174, Time since last communication: 0.000s
I20240503 02:12:24.921864 26136 raft_consensus.cc:1261] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 175. Preceding OpId from leader: term: 1 index: 176. (index mismatch)
I20240503 02:12:24.922453 28770 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 174, Time since last communication: 0.000s
I20240503 02:12:24.924187 26529 raft_consensus.cc:1261] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 169. Preceding OpId from leader: term: 1 index: 176. (index mismatch)
I20240503 02:12:24.924762 28770 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 169, Time since last communication: 0.000s
I20240503 02:12:24.926800 28770 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Committing config change with OpId 1.176: config changed from index 168 to 176, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.927657 26525 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.310: config changed from index 305 to 310, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.932493 26136 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.176: config changed from index 168 to 176, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.948810 26653 raft_consensus.cc:1203] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.174->[1.175-1.175] Dedup: 1.175->[]
I20240503 02:12:24.950358 25853 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 740218c49aa84ee28f875f2cfcee01b8 with cas_config_opid_index 168: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:24.952222 26653 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.176: config changed from index 168 to 176, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.954573 25866 catalog_manager.cc:5484] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb reported cstate change: config changed from index 168 to 176, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:24.960137 27593 raft_consensus.cc:1261] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 175. Preceding OpId from leader: term: 1 index: 176. (index mismatch)
I20240503 02:12:24.960497 25988 tablet_service.cc:1514] Processing DeleteTablet for tablet 740218c49aa84ee28f875f2cfcee01b8 with delete_type TABLET_DATA_TOMBSTONED (TS 3bb103b2fd2344e7b55caf7389acf221 not found in new config with opid_index 176) from {username='slave'} at 127.0.0.1:43226
I20240503 02:12:24.961534 26528 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.176: config changed from index 168 to 176, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:24.961956 28770 consensus_queue.cc:1035] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 174, Time since last communication: 0.000s
I20240503 02:12:24.964619 27589 raft_consensus.cc:2932] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.176: config changed from index 168 to 176, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:24.986752 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 83084.8us
I20240503 02:12:24.991963 26524 raft_consensus.cc:1203] T 740218c49aa84ee28f875f2cfcee01b8 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.169->[1.170-1.174] Dedup: 1.174->[]
I20240503 02:12:24.996007 25866 catalog_manager.cc:5484] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 reported cstate change: config changed from index 305 to 310, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "ae8b30bcfa5b425782a95374787dd230" committed_config { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:25.000463 26378 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d with delete_type TABLET_DATA_TOMBSTONED (TS cff7b7a19fc64db5849391b1d8e10cce not found in new config with opid_index 310) from {username='slave'} at 127.0.0.1:48358
I20240503 02:12:25.008128 27593 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.310: config changed from index 305 to 310, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:25.011467 29061 tablet_replica.cc:330] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce: stopping tablet replica
I20240503 02:12:25.001492 26003 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.310: config changed from index 305 to 310, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:25.012526 29061 raft_consensus.cc:2229] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.017640 29059 tablet_replica.cc:330] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221: stopping tablet replica
I20240503 02:12:25.018301 29059 raft_consensus.cc:2229] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.018452 29061 pending_rounds.cc:62] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce: Trying to abort 1 pending ops.
I20240503 02:12:25.018817 29059 pending_rounds.cc:62] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221: Trying to abort 1 pending ops.
I20240503 02:12:25.019003 29059 pending_rounds.cc:68] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221: Aborting op as it isn't in flight: id { term: 1 index: 175 } timestamp: 7023420804743065600 op_type: WRITE_OP write_request { tablet_id: "740218c49aa84ee28f875f2cfcee01b8" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\004EX\002\014\326b\246\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""9d4f697c112cb5d9" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023420804596559872 authz_token { token_data: "\010\227\222\321\261\006\"3\n\005slave\022*\n d4abb1a36f9a48a3b886746b377881e5\020\001\030\001 \001(\001" signature: "<redacted>""\343e\374x\235\332A\272\301Ns\304\317]\234\356\344\246h\332<m\3446P\022\3757\277\223\\\237II\023\303\027\r\321\377\017\326\033N^\034\366\203\206\227\200\263\300.8w\322D\236\352\243\245x\375" signing_key_seq_num: 0 } } request_id { client_id: "6a17428734c64e988ecc470e9d42344e" seq_no: 480 first_incomplete_seq_no: 480 attempt_no: 0 }
I20240503 02:12:25.020589 29059 raft_consensus.cc:2258] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.023257 29059 ts_tablet_manager.cc:1875] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:25.024721 29061 pending_rounds.cc:68] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce: Aborting op as it isn't in flight: id { term: 1 index: 309 } timestamp: 7023420803557367808 op_type: WRITE_OP write_request { tablet_id: "7eb1a7e7a1a440c1a826ac2f2e27db2d" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000u\344W?\320\224\243]\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5473b182ef15f23d" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023420803305316352 authz_token { token_data: "\010\227\222\321\261\006\"3\n\005slave\022*\n 139fcfbed25b4601a8192d5ddba81e27\020\001\030\001 \001(\001" signature: "<redacted>""\233\335\201t\275\224j\324\274\025\021\315\327\221\3232\277\337\200/\026\234\023\223pp]\315h\200|B\204\346\021kM\353\345\014\3019\024\001?\210\372\265\004:\327W\370+\3766\320\203\201\361E\345\245\313" signing_key_seq_num: 0 } } request_id { client_id: "6d4bb856b5db4024baf5e69853284c81" seq_no: 796 first_incomplete_seq_no: 796 attempt_no: 0 }
I20240503 02:12:25.026149 29061 raft_consensus.cc:2258] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.038331 26138 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.310: config changed from index 305 to 310, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:25.062829 29061 ts_tablet_manager.cc:1875] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:25.073467 29059 ts_tablet_manager.cc:1888] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.175
I20240503 02:12:25.103005 29059 log.cc:1196] T 740218c49aa84ee28f875f2cfcee01b8 P 3bb103b2fd2344e7b55caf7389acf221: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/740218c49aa84ee28f875f2cfcee01b8
I20240503 02:12:25.104267 25851 catalog_manager.cc:4830] TS 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): tablet 740218c49aa84ee28f875f2cfcee01b8 (table rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]) successfully deleted
I20240503 02:12:25.105482 29061 ts_tablet_manager.cc:1888] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.309
I20240503 02:12:25.115954 29061 log.cc:1196] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P cff7b7a19fc64db5849391b1d8e10cce: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/7eb1a7e7a1a440c1a826ac2f2e27db2d
W20240503 02:12:25.131642 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 77126.1us
I20240503 02:12:25.154744 25853 catalog_manager.cc:4830] TS cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675): tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d (table rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]) successfully deleted
W20240503 02:12:25.159999 26595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 102089us
I20240503 02:12:25.175928 27592 raft_consensus.cc:1203] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.250->[1.251-1.281] Dedup: 1.271->[1.272-1.281]
W20240503 02:12:25.281311 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 88130.1us
I20240503 02:12:25.370330 28560 raft_consensus.cc:1050] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e: attempting to promote NON_VOTER 0a801f0897884f108cdb89434b18628a to VOTER
I20240503 02:12:25.371722 28560 consensus_queue.cc:237] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 1.282, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:25.375699 27589 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a [term 1 LEARNER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20240503 02:12:25.376904 28560 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 281, Time since last communication: 0.000s
I20240503 02:12:25.383291 26137 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20240503 02:12:25.390769 28560 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 281, Time since last communication: 0.000s
I20240503 02:12:25.427599 26392 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20240503 02:12:25.428814 26269 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
W20240503 02:12:25.446137 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 383697us
I20240503 02:12:25.516022 26653 consensus_queue.cc:237] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 3.253, Last appended by leader: 234, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:25.519387 26397 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 253. Preceding OpId from leader: term: 3 index: 254. (index mismatch)
I20240503 02:12:25.519471 26267 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 253. Preceding OpId from leader: term: 3 index: 254. (index mismatch)
I20240503 02:12:25.520196 29068 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 252, Time since last communication: 0.000s
I20240503 02:12:25.520691 29068 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 252, Time since last communication: 0.000s
I20240503 02:12:25.524830 29028 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 LEADER]: Committing config change with OpId 3.254: config changed from index 251 to 254, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:25.529143 26397 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Committing config change with OpId 3.254: config changed from index 251 to 254, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:25.546061 25854 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4a864ac954e046eda8925f5daf4a0f1f with cas_config_opid_index 251: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:25.546839 26267 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Committing config change with OpId 3.254: config changed from index 251 to 254, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:25.547183 25866 catalog_manager.cc:5484] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: config changed from index 251 to 254, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New cstate: current_term: 3 leader_uuid: "ae8b30bcfa5b425782a95374787dd230" committed_config { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:25.445647 29038 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 281, Time since last communication: 0.003s
I20240503 02:12:25.561753 28501 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 281, Time since last communication: 0.007s
I20240503 02:12:25.578532 26657 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20240503 02:12:25.578158 26265 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 251 to 283, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:25.583088 26137 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 253. Preceding OpId from leader: term: 3 index: 254. (index mismatch)
I20240503 02:12:25.607455 25859 catalog_manager.cc:5484] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb reported cstate change: config changed from index 251 to 283, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:25.614730 29039 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 281, Time since last communication: 0.024s
I20240503 02:12:25.585857 29038 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 LEADER]: Committing config change with OpId 1.283: config changed from index 251 to 283, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:25.658003 26392 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 251 to 283, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:25.673239 27593 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 251 to 283, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:25.685882 26136 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 251 to 283, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:25.777606 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 44355.7us
I20240503 02:12:25.806555 27589 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 253. Preceding OpId from leader: term: 3 index: 254. (index mismatch)
I20240503 02:12:25.836833 29066 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 252, Time since last communication: 0.000s
I20240503 02:12:25.876724 26138 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Committing config change with OpId 3.254: config changed from index 251 to 254, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:25.931370 26654 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 251 to 283, 0a801f0897884f108cdb89434b18628a (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:25.942662 29068 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 252, Time since last communication: 0.000s
I20240503 02:12:25.952373 25988 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a864ac954e046eda8925f5daf4a0f1f with delete_type TABLET_DATA_TOMBSTONED (TS 3bb103b2fd2344e7b55caf7389acf221 not found in new config with opid_index 254) from {username='slave'} at 127.0.0.1:43226
I20240503 02:12:25.968228 29104 tablet_replica.cc:330] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221: stopping tablet replica
I20240503 02:12:25.995608 26523 consensus_queue.cc:237] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 1.287, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:12:25.998028 26269 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 287. Preceding OpId from leader: term: 1 index: 288. (index mismatch)
I20240503 02:12:25.998821 29038 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 286, Time since last communication: 0.000s
I20240503 02:12:26.007565 26397 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 286. Preceding OpId from leader: term: 1 index: 288. (index mismatch)
I20240503 02:12:26.008111 29092 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 286, Time since last communication: 0.000s
I20240503 02:12:26.012908 29092 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 LEADER]: Committing config change with OpId 1.288: config changed from index 283 to 288, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:26.034557 25854 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 508ee4b913dd40b49d50be3fa4cdd6c4 with cas_config_opid_index 283: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:26.035524 26267 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 283 to 288, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:26.036422 25866 catalog_manager.cc:5484] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e reported cstate change: config changed from index 283 to 288, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" committed_config { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:26.036612 26397 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 283 to 288, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:26.041812 26117 tablet_service.cc:1514] Processing DeleteTablet for tablet 508ee4b913dd40b49d50be3fa4cdd6c4 with delete_type TABLET_DATA_TOMBSTONED (TS 2cf1cac003b44643a2d4de1204aacf0f not found in new config with opid_index 288) from {username='slave'} at 127.0.0.1:54956
I20240503 02:12:26.114025 29104 raft_consensus.cc:2229] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:26.118276 27586 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 3 FOLLOWER]: Committing config change with OpId 3.254: config changed from index 251 to 254, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
I20240503 02:12:26.274616 29104 pending_rounds.cc:62] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221: Trying to abort 1 pending ops.
I20240503 02:12:26.274824 29104 pending_rounds.cc:68] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221: Aborting op as it isn't in flight: id { term: 3 index: 253 } timestamp: 7023420805351411712 op_type: WRITE_OP write_request { tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\367\233\3170\230n\255S\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""c7b2a3b4423304bd" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023420805291798528 authz_token { token_data: "\010\227\222\321\261\006\"3\n\005slave\022*\n 139fcfbed25b4601a8192d5ddba81e27\020\001\030\001 \001(\001" signature: "<redacted>""\233\335\201t\275\224j\324\274\025\021\315\327\221\3232\277\337\200/\026\234\023\223pp]\315h\200|B\204\346\021kM\353\345\014\3019\024\001?\210\372\265\004:\327W\370+\3766\320\203\201\361E\345\245\313" signing_key_seq_num: 0 } } request_id { client_id: "6d4bb856b5db4024baf5e69853284c81" seq_no: 800 first_incomplete_seq_no: 800 attempt_no: 0 }
I20240503 02:12:26.275950 29104 raft_consensus.cc:2258] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:26.277130 29104 ts_tablet_manager.cc:1875] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:12:26.344996 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 227998us
W20240503 02:12:26.401221 27593 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.036s sys 0.040s
W20240503 02:12:26.401340 27593 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.724s user 0.036s sys 0.040s
I20240503 02:12:26.401648 27594 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 288. (index mismatch)
I20240503 02:12:26.402324 28501 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 283, Time since last communication: 0.000s
I20240503 02:12:26.464931 26657 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 286. Preceding OpId from leader: term: 1 index: 288. (index mismatch)
I20240503 02:12:26.472762 28560 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 285, Time since last communication: 0.000s
I20240503 02:12:26.482532 27593 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 283 to 288, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:26.585578 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 66979.8us
I20240503 02:12:26.656453 26658 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 283 to 288, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } }
W20240503 02:12:26.725585 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 76438.6us
I20240503 02:12:26.740258 29109 tablet_replica.cc:330] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f: stopping tablet replica
I20240503 02:12:26.740543 29109 raft_consensus.cc:2229] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:26.741034 29109 raft_consensus.cc:2258] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:26.743858 26117 thread.cc:636] tablet-delete [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.001s sys 0.012s
W20240503 02:12:26.744103 26117 thread.cc:602] tablet-delete [worker] (thread pool) Time spent starting thread: real 0.702s user 0.001s sys 0.012s
I20240503 02:12:26.773031 29109 ts_tablet_manager.cc:1875] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:26.779693 29104 ts_tablet_manager.cc:1888] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.253
I20240503 02:12:26.779944 29104 log.cc:1196] T 4a864ac954e046eda8925f5daf4a0f1f P 3bb103b2fd2344e7b55caf7389acf221: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/4a864ac954e046eda8925f5daf4a0f1f
I20240503 02:12:26.782835 25851 catalog_manager.cc:4830] TS 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): tablet 4a864ac954e046eda8925f5daf4a0f1f (table rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]) successfully deleted
W20240503 02:12:26.933281 25944 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 164894us
W20240503 02:12:26.969681 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 54365.5us
I20240503 02:12:27.156850 29109 ts_tablet_manager.cc:1888] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.286
I20240503 02:12:27.157111 29109 log.cc:1196] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 2cf1cac003b44643a2d4de1204aacf0f: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/508ee4b913dd40b49d50be3fa4cdd6c4
I20240503 02:12:27.158093 29109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54956 (request call id 17) took 1117 ms. Trace:
I20240503 02:12:27.158267 29109 rpcz_store.cc:276] 0503 02:12:26.040997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:26.041698 (+ 701us) service_pool.cc:225] Handling call
0503 02:12:26.740068 (+698370us) ts_tablet_manager.cc:1120] Deleting tablet 508ee4b913dd40b49d50be3fa4cdd6c4
0503 02:12:26.740079 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0503 02:12:26.894652 (+154573us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:27.156670 (+262018us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:27.158039 (+ 1369us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1,"tablet-delete.queue_time_us":698084,"thread_start_us":702209,"threads_started":1}
I20240503 02:12:27.159030 25851 catalog_manager.cc:4830] TS 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): tablet 508ee4b913dd40b49d50be3fa4cdd6c4 (table rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]) successfully deleted
W20240503 02:12:27.478101 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 97867.3us
W20240503 02:12:28.024597 26202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 94483.4us
W20240503 02:12:28.408304 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 121252us
W20240503 02:12:28.574482 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 48144.7us
W20240503 02:12:28.934324 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 88383.3us
I20240503 02:12:29.141558 26117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:29.151285 25988 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 02:12:29.261288 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 295573us
I20240503 02:12:29.272766 27573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:29.274685 26638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 02:12:29.332002 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 44268.4us
I20240503 02:12:29.335572 26509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:29.350183 26378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:29.575091 26247 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:12:30.359817 26465 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 70450.7us
W20240503 02:12:30.361300 26202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 60277.6us
W20240503 02:12:30.608873 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 99469us
W20240503 02:12:31.333295 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 56522.9us
W20240503 02:12:31.463259 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 72240.6us
I20240503 02:12:31.596042 27573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:31.627264 26117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:31.648617 26638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 02:12:31.676065 25941 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 50447.9us
I20240503 02:12:31.770351 25988 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:31.816888 26247 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:32.031276 26378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:12:32.057235 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 88424.4us
I20240503 02:12:32.175339 26509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 02:12:32.178157 26202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 108988us
W20240503 02:12:32.707206 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 56285.5us
W20240503 02:12:33.036111 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 188601us
W20240503 02:12:33.428283 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 61649.9us
W20240503 02:12:33.562130 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 60011.1us
W20240503 02:12:33.632220 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 54125.7us
W20240503 02:12:33.730363 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 46100.8us
W20240503 02:12:34.190387 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 51219.4us
W20240503 02:12:34.268492 26202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 115245us
W20240503 02:12:34.475261 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 60365.7us
W20240503 02:12:34.667246 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 182751us
W20240503 02:12:34.890056 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 93849.8us
W20240503 02:12:35.602968 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 256987us
W20240503 02:12:35.615702 26595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 110074us
W20240503 02:12:35.645962 26074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 43859.9us
W20240503 02:12:35.661715 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 60138.5us
W20240503 02:12:35.710861 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 65712.4us
W20240503 02:12:35.804198 25943 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 93945.9us
W20240503 02:12:35.804441 25941 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 62755.1us
W20240503 02:12:35.826391 26595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 52142.6us
W20240503 02:12:35.915659 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 72747.6us
W20240503 02:12:36.246775 26074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 46752.6us
W20240503 02:12:37.010579 26074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 44878.6us
W20240503 02:12:37.044243 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 46708.5us
I20240503 02:12:37.448381 27573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:37.481756 26509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:37.484138 26638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:37.535008 25988 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 02:12:37.636766 26061 kernel_stack_watchdog.cc:198] Thread 29464 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 263ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7ff14c9dc980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7ff14964ad11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7ff14966b7ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7ff1496673ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7ff14966c4ff kudu::ThreadPool::Submit() at ??:0
@ 0x7ff15480be87 kudu::log::Log::AppendThread::Wake() at ??:0
@ 0x7ff154818fbc kudu::log::Log::AsyncAppend() at ??:0
@ 0x7ff154819995 kudu::log::Log::AsyncAppendReplicates() at ??:0
@ 0x7ff154e1747d kudu::consensus::LogCache::AppendOperations() at ??:0
@ 0x7ff154db8b1b kudu::consensus::PeerMessageQueue::AppendOperations() at ??:0
@ 0x7ff154db812f kudu::consensus::PeerMessageQueue::AppendOperation() at ??:0
@ 0x7ff154e6b2d6 kudu::consensus::RaftConsensus::AppendNewRoundToQueueUnlocked() at ??:0
@ 0x7ff154e6c13a kudu::consensus::RaftConsensus::Replicate() at ??:0
W20240503 02:12:37.757011 26074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 289519us
W20240503 02:12:37.811281 29464 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.002s sys 0.007s
W20240503 02:12:37.811393 29464 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s user 0.002s sys 0.007s
W20240503 02:12:37.947813 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 69819.4us
W20240503 02:12:37.953137 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 85689.8us
W20240503 02:12:37.959755 26074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 50133.5us
I20240503 02:12:37.987571 26247 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:38.047502 26378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:12:38.084681 26202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 43752.2us
W20240503 02:12:38.087541 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 47603.4us
W20240503 02:12:38.242908 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 106910us
W20240503 02:12:38.319139 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 75977.2us
I20240503 02:12:38.701476 26117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 02:12:39.280869 26205 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 139621us
W20240503 02:12:39.533977 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 85355.4us
W20240503 02:12:39.706285 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 86011.4us
W20240503 02:12:39.868813 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 88939.5us
W20240503 02:12:39.876979 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 258695us
W20240503 02:12:39.998778 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 101839us
W20240503 02:12:40.380405 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 362109us
W20240503 02:12:40.473106 26138 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.000s sys 0.006s
W20240503 02:12:40.473227 26138 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.516s user 0.000s sys 0.006s
W20240503 02:12:40.591989 29512 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.065s sys 0.154s
W20240503 02:12:40.592084 29512 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.619s user 0.065s sys 0.154s
W20240503 02:12:40.670436 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 45714.6us
W20240503 02:12:40.935510 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 50190.4us
W20240503 02:12:42.151064 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 44551us
W20240503 02:12:42.177760 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 87110.3us
I20240503 02:12:42.841495 27573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:42.938706 25988 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:42.969431 26117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:43.136960 26638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 02:12:43.151412 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 50222us
W20240503 02:12:43.261709 26202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 71799.9us
I20240503 02:12:43.294029 26247 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:43.407125 26378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:43.461463 26509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 02:12:43.901822 26072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:34279, user_credentials={real_user=slave}} blocked reactor thread for 48762us
W20240503 02:12:44.001243 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 68753.4us
W20240503 02:12:44.131984 26332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:34143, user_credentials={real_user=slave}} blocked reactor thread for 47061.2us
W20240503 02:12:44.418038 26592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:39753, user_credentials={real_user=slave}} blocked reactor thread for 60098us
W20240503 02:12:45.474223 25944 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:40369, user_credentials={real_user=slave}} blocked reactor thread for 45588.8us
W20240503 02:12:45.518242 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 61822.9us
W20240503 02:12:45.667061 26464 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45999, user_credentials={real_user=slave}} blocked reactor thread for 49903.4us
W20240503 02:12:45.675768 26075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:38347, user_credentials={real_user=slave}} blocked reactor thread for 88964.7us
W20240503 02:12:45.743896 26074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:43675, user_credentials={real_user=slave}} blocked reactor thread for 154613us
I20240503 02:12:45.889569 27573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:45.918398 26117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:45.950165 25988 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:45.955425 26247 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:45.978696 26378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:46.017949 26638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:46.021348 26509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b8f9a407d2264aa2ba34e0ddd69ece11 | 127.23.104.254:40709 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.23.104.212:46007 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
0a801f0897884f108cdb89434b18628a | 127.23.104.199:34279 | HEALTHY | <none> | 0 | 0
2cf1cac003b44643a2d4de1204aacf0f | 127.23.104.194:39753 | HEALTHY | <none> | 4 | 0
3bb103b2fd2344e7b55caf7389acf221 | 127.23.104.193:34143 | HEALTHY | <none> | 2 | 0
7c384005cb3041a0bd70e163473b6acb | 127.23.104.195:45999 | HEALTHY | <none> | 3 | 0
ae8b30bcfa5b425782a95374787dd230 | 127.23.104.198:40369 | HEALTHY | <none> | 2 | 0
b2835e59d40a48aea53c5dedb3bd1e9e | 127.23.104.197:38347 | HEALTHY | <none> | 2 | 0
cff7b7a19fc64db5849391b1d8e10cce | 127.23.104.196:43675 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.23.104.193 | experimental | 127.23.104.193:34143
local_ip_for_outbound_sockets | 127.23.104.194 | experimental | 127.23.104.194:39753
local_ip_for_outbound_sockets | 127.23.104.195 | experimental | 127.23.104.195:45999
local_ip_for_outbound_sockets | 127.23.104.196 | experimental | 127.23.104.196:43675
local_ip_for_outbound_sockets | 127.23.104.197 | experimental | 127.23.104.197:38347
local_ip_for_outbound_sockets | 127.23.104.198 | experimental | 127.23.104.198:40369
local_ip_for_outbound_sockets | 127.23.104.199 | experimental | 127.23.104.199:34279
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.104.193:34143
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.23.104.194:39753
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.104.195:45999
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.23.104.196:43675
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.23.104.197:38347
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.23.104.198:40369
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.23.104.199:34279
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.23.104.212:46007 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 11
Median | 11
Third Quartile | 12
Maximum | 12
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 02:12:46.100796 23971 log_verifier.cc:126] Checking tablet 041ffbedf2c04dbea4661b4c33f78ac1
I20240503 02:12:46.611744 23971 log_verifier.cc:177] Verified matching terms for 530 ops in tablet 041ffbedf2c04dbea4661b4c33f78ac1
I20240503 02:12:46.613270 23971 log_verifier.cc:126] Checking tablet 1162f9075da248d0a919db2eb45aa949
I20240503 02:12:47.075984 23971 log_verifier.cc:177] Verified matching terms for 501 ops in tablet 1162f9075da248d0a919db2eb45aa949
I20240503 02:12:47.077214 23971 log_verifier.cc:126] Checking tablet 21e7ddb56be1499b94920ebd252595ce
I20240503 02:12:47.542261 23971 log_verifier.cc:177] Verified matching terms for 514 ops in tablet 21e7ddb56be1499b94920ebd252595ce
I20240503 02:12:47.543503 23971 log_verifier.cc:126] Checking tablet 39d3ba2cbbad40d6a232e534c65d2d74
I20240503 02:12:47.806372 23971 log_verifier.cc:177] Verified matching terms for 247 ops in tablet 39d3ba2cbbad40d6a232e534c65d2d74
I20240503 02:12:47.806948 23971 log_verifier.cc:126] Checking tablet 495c99a8850d4ec6b52a89bec172bab0
I20240503 02:12:48.318953 23971 log_verifier.cc:177] Verified matching terms for 518 ops in tablet 495c99a8850d4ec6b52a89bec172bab0
I20240503 02:12:48.320432 23971 log_verifier.cc:126] Checking tablet 4a864ac954e046eda8925f5daf4a0f1f
I20240503 02:12:48.941967 23971 log_verifier.cc:177] Verified matching terms for 495 ops in tablet 4a864ac954e046eda8925f5daf4a0f1f
I20240503 02:12:48.943127 23971 log_verifier.cc:126] Checking tablet 508ee4b913dd40b49d50be3fa4cdd6c4
I20240503 02:12:49.471469 23971 log_verifier.cc:177] Verified matching terms for 519 ops in tablet 508ee4b913dd40b49d50be3fa4cdd6c4
I20240503 02:12:49.472728 23971 log_verifier.cc:126] Checking tablet 5332ecb1e4454eadb47cc721e2fbc487
I20240503 02:12:49.517117 23971 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 5332ecb1e4454eadb47cc721e2fbc487
I20240503 02:12:49.517210 23971 log_verifier.cc:126] Checking tablet 740218c49aa84ee28f875f2cfcee01b8
I20240503 02:12:49.802024 23971 log_verifier.cc:177] Verified matching terms for 269 ops in tablet 740218c49aa84ee28f875f2cfcee01b8
I20240503 02:12:49.802589 23971 log_verifier.cc:126] Checking tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d
I20240503 02:12:50.314710 23971 log_verifier.cc:177] Verified matching terms for 566 ops in tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d
I20240503 02:12:50.315980 23971 log_verifier.cc:126] Checking tablet c12cc2e0fb09431e95baae55803f2184
I20240503 02:12:50.585999 23971 log_verifier.cc:177] Verified matching terms for 266 ops in tablet c12cc2e0fb09431e95baae55803f2184
I20240503 02:12:50.586654 23971 log_verifier.cc:126] Checking tablet c27502976d474b938cce2235e5784f79
I20240503 02:12:50.877133 23971 log_verifier.cc:177] Verified matching terms for 280 ops in tablet c27502976d474b938cce2235e5784f79
I20240503 02:12:50.877808 23971 log_verifier.cc:126] Checking tablet c3b6c051161d4f9ba9d1d38a06a4c633
I20240503 02:12:51.319154 23971 log_verifier.cc:177] Verified matching terms for 488 ops in tablet c3b6c051161d4f9ba9d1d38a06a4c633
I20240503 02:12:51.320282 23971 log_verifier.cc:126] Checking tablet d972dd11fcd9484f9a59f63d89f1e6e2
I20240503 02:12:51.818567 23971 log_verifier.cc:177] Verified matching terms for 522 ops in tablet d972dd11fcd9484f9a59f63d89f1e6e2
I20240503 02:12:51.820044 23971 log_verifier.cc:126] Checking tablet e866b4f1b338419ab1025287b2aaac5e
I20240503 02:12:52.097481 23971 log_verifier.cc:177] Verified matching terms for 277 ops in tablet e866b4f1b338419ab1025287b2aaac5e
I20240503 02:12:52.098109 23971 log_verifier.cc:126] Checking tablet fa2fbede85374d1492edeaafef243747
I20240503 02:12:52.395027 23971 log_verifier.cc:177] Verified matching terms for 269 ops in tablet fa2fbede85374d1492edeaafef243747
I20240503 02:12:52.406558 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.200:0
--local_ip_for_outbound_sockets=127.23.104.200
--webserver_interface=127.23.104.200
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--builtin_ntp_servers=127.23.104.212:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:12:52.769193 29797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:52.769469 29797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:52.769588 29797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:52.777242 29797 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:52.777355 29797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:52.777508 29797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.200
I20240503 02:12:52.786314 29797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.23.104.200
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40709
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.200
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:52.787640 29797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:52.789477 29797 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:52.802451 29803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:52.802918 29804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:52.803642 29806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:52.804389 29797 server_base.cc:1034] running on GCE node
I20240503 02:12:52.804900 29797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:52.805717 29797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:52.806905 29797 hybrid_clock.cc:648] HybridClock initialized: now 1714702372806845 us; error 45 us; skew 500 ppm
I20240503 02:12:52.807168 29797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:52.809360 29797 webserver.cc:460] Webserver started at http://127.23.104.200:42733/ using document root <none> and password file <none>
I20240503 02:12:52.810469 29797 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:52.810566 29797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:52.810848 29797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:52.813261 29797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "1b4b70e4a9d54c909b06663e702d4964"
format_stamp: "Formatted at 2024-05-03 02:12:52 on dist-test-slave-38ht"
I20240503 02:12:52.813970 29797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "1b4b70e4a9d54c909b06663e702d4964"
format_stamp: "Formatted at 2024-05-03 02:12:52 on dist-test-slave-38ht"
I20240503 02:12:52.818331 29797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:12:52.821637 29812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:52.822906 29797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:52.823079 29797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "1b4b70e4a9d54c909b06663e702d4964"
format_stamp: "Formatted at 2024-05-03 02:12:52 on dist-test-slave-38ht"
I20240503 02:12:52.823251 29797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:52.842782 29797 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:52.843685 29797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:52.844074 29797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:52.847060 29797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:52.848919 29797 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:52.849020 29797 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 02:12:52.849124 29797 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:52.849195 29797 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:52.896103 29797 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.200:34857
I20240503 02:12:52.896147 29924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.200:34857 every 8 connection(s)
I20240503 02:12:52.897347 29797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:12:52.900161 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 29797
I20240503 02:12:52.900369 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:12:52.911475 29925 heartbeater.cc:347] Connected to a master server at 127.23.104.254:40709
I20240503 02:12:52.911866 29925 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:52.912837 29925 heartbeater.cc:511] Master 127.23.104.254:40709 requested a full tablet report, sending...
I20240503 02:12:52.914716 25866 ts_manager.cc:194] Registered new tserver with Master: 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857)
I20240503 02:12:52.915431 25866 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.200:56831
I20240503 02:12:53.416738 27573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:53.428265 26117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:53.461417 29860 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:53.488571 25988 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:53.506095 26247 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:53.506805 26638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:53.524233 26509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:53.546574 26378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:53.724948 27573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:53.740243 25988 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:53.741458 26117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:53.747103 26247 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:53.748749 29860 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:53.765399 26638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:53.782276 26509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:53.797212 26378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:53.900745 26393 consensus_queue.cc:237] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 522, Committed index: 522, Last appended: 1.522, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:53.904125 26008 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 522. Preceding OpId from leader: term: 1 index: 523. (index mismatch)
I20240503 02:12:53.904294 26138 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 522. Preceding OpId from leader: term: 1 index: 523. (index mismatch)
I20240503 02:12:53.905021 29787 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 523, Last known committed idx: 522, Time since last communication: 0.000s
I20240503 02:12:53.905736 29787 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 523, Last known committed idx: 522, Time since last communication: 0.000s
W20240503 02:12:53.911262 26334 consensus_peers.cc:485] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce -> Peer 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857): Couldn't send request to peer 1b4b70e4a9d54c909b06663e702d4964. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d972dd11fcd9484f9a59f63d89f1e6e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:53.912521 27594 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 522. Preceding OpId from leader: term: 1 index: 523. (index mismatch)
I20240503 02:12:53.912776 26654 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 522. Preceding OpId from leader: term: 1 index: 523. (index mismatch)
I20240503 02:12:53.912415 29985 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 1 LEADER]: Committing config change with OpId 1.523: config changed from index 105 to 523, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:53.913079 30022 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 523, Last known committed idx: 522, Time since last communication: 0.000s
I20240503 02:12:53.913483 29787 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 523, Last known committed idx: 522, Time since last communication: 0.000s
I20240503 02:12:53.913292 26008 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.523: config changed from index 105 to 523, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:53.917371 26138 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.523: config changed from index 105 to 523, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:53.918109 29925 heartbeater.cc:503] Master 127.23.104.254:40709 was elected leader, sending a full tablet report...
I20240503 02:12:53.922902 25859 catalog_manager.cc:5484] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 reported cstate change: config changed from index 105 to 523, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "cff7b7a19fc64db5849391b1d8e10cce" committed_config { opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:53.934571 27594 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.523: config changed from index 105 to 523, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:53.940694 26654 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.523: config changed from index 105 to 523, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:53.954208 26138 consensus_queue.cc:237] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 266, Committed index: 266, Last appended: 1.266, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:53.956817 27594 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20240503 02:12:53.958676 29794 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I20240503 02:12:53.965066 26008 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20240503 02:12:53.965147 26524 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20240503 02:12:53.965636 29794 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I20240503 02:12:53.965876 30042 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I20240503 02:12:53.968573 30039 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Committing config change with OpId 1.267: config changed from index 143 to 267, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:53.969096 26008 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index 143 to 267, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:53.969099 26524 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index 143 to 267, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:53.973309 27594 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index 143 to 267, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:53.973811 25864 catalog_manager.cc:5484] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 reported cstate change: config changed from index 143 to 267, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:53.976042 26393 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20240503 02:12:53.977452 30039 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I20240503 02:12:53.983999 26393 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index 143 to 267, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
W20240503 02:12:53.988263 26074 consensus_peers.cc:485] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f -> Peer 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857): Couldn't send request to peer 1b4b70e4a9d54c909b06663e702d4964. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c12cc2e0fb09431e95baae55803f2184. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:53.996955 26524 consensus_queue.cc:237] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 519, Committed index: 519, Last appended: 1.519, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 520 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:54.000411 27594 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 519. Preceding OpId from leader: term: 1 index: 520. (index mismatch)
I20240503 02:12:54.001067 29798 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 520, Last known committed idx: 519, Time since last communication: 0.000s
I20240503 02:12:54.008808 26654 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 519. Preceding OpId from leader: term: 1 index: 520. (index mismatch)
I20240503 02:12:54.009415 30053 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 520, Last known committed idx: 519, Time since last communication: 0.000s
I20240503 02:12:54.009760 26393 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 519. Preceding OpId from leader: term: 1 index: 520. (index mismatch)
I20240503 02:12:54.010303 30053 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 520, Last known committed idx: 519, Time since last communication: 0.000s
I20240503 02:12:54.011173 26264 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 519. Preceding OpId from leader: term: 1 index: 520. (index mismatch)
I20240503 02:12:54.011713 30053 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 520, Last known committed idx: 519, Time since last communication: 0.000s
I20240503 02:12:54.015079 29798 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 LEADER]: Committing config change with OpId 1.520: config changed from index 288 to 520, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 520 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.020524 27594 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.520: config changed from index 288 to 520, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 520 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.021972 25858 catalog_manager.cc:5484] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e reported cstate change: config changed from index 288 to 520, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" committed_config { opid_index: 520 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:54.042716 26464 consensus_peers.cc:485] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e -> Peer 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857): Couldn't send request to peer 1b4b70e4a9d54c909b06663e702d4964. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 508ee4b913dd40b49d50be3fa4cdd6c4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:54.043963 26654 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.520: config changed from index 288 to 520, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 520 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.044987 26264 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.520: config changed from index 288 to 520, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 520 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.059998 26393 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.520: config changed from index 288 to 520, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 520 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.073174 26654 consensus_queue.cc:237] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 566, Committed index: 566, Last appended: 1.566, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 567 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:54.076303 26008 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 566. Preceding OpId from leader: term: 1 index: 567. (index mismatch)
I20240503 02:12:54.076797 27594 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 566. Preceding OpId from leader: term: 1 index: 567. (index mismatch)
I20240503 02:12:54.076898 30051 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 567, Last known committed idx: 566, Time since last communication: 0.000s
I20240503 02:12:54.078704 30051 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 567, Last known committed idx: 566, Time since last communication: 0.000s
I20240503 02:12:54.086665 30051 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 1 LEADER]: Committing config change with OpId 1.567: config changed from index 310 to 567, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 567 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.087400 27594 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.567: config changed from index 310 to 567, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 567 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.090559 26524 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 566. Preceding OpId from leader: term: 1 index: 567. (index mismatch)
I20240503 02:12:54.091406 26138 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 566. Preceding OpId from leader: term: 1 index: 567. (index mismatch)
I20240503 02:12:54.091735 30066 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 567, Last known committed idx: 566, Time since last communication: 0.000s
I20240503 02:12:54.092175 29237 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 567, Last known committed idx: 566, Time since last communication: 0.000s
I20240503 02:12:54.098188 25858 catalog_manager.cc:5484] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a reported cstate change: config changed from index 310 to 567, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "ae8b30bcfa5b425782a95374787dd230" committed_config { opid_index: 567 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
W20240503 02:12:54.099355 26594 consensus_peers.cc:485] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 -> Peer 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857): Couldn't send request to peer 1b4b70e4a9d54c909b06663e702d4964. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7eb1a7e7a1a440c1a826ac2f2e27db2d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:54.106267 26524 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.567: config changed from index 310 to 567, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 567 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.114264 26008 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.567: config changed from index 310 to 567, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 567 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.113566 26138 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.567: config changed from index 310 to 567, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 567 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.131490 26008 consensus_queue.cc:237] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:54.134601 26524 raft_consensus.cc:1261] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:54.134582 26654 raft_consensus.cc:1261] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:54.135177 30078 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:54.135607 29692 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:12:54.145804 25943 consensus_peers.cc:485] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 -> Peer 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857): Couldn't send request to peer 1b4b70e4a9d54c909b06663e702d4964. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5332ecb1e4454eadb47cc721e2fbc487. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:54.148370 30078 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.152576 27594 raft_consensus.cc:1261] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:54.152997 26264 raft_consensus.cc:1261] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:54.154889 30079 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:54.160495 27594 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.162683 30084 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:54.167328 25858 catalog_manager.cc:5484] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a reported cstate change: config changed from index -1 to 2, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "3bb103b2fd2344e7b55caf7389acf221" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.191053 26524 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.193406 26264 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.199573 26654 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:54.284946 30114 ts_tablet_manager.cc:902] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964: Initiating tablet copy from peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675)
I20240503 02:12:54.286396 30114 tablet_copy_client.cc:308] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:43675
I20240503 02:12:54.291833 26419 tablet_copy_service.cc:140] P cff7b7a19fc64db5849391b1d8e10cce: Received BeginTabletCopySession request for tablet d972dd11fcd9484f9a59f63d89f1e6e2 from peer 1b4b70e4a9d54c909b06663e702d4964 ({username='slave'} at 127.23.104.200:49621)
I20240503 02:12:54.291936 26419 tablet_copy_service.cc:161] P cff7b7a19fc64db5849391b1d8e10cce: Beginning new tablet copy session on tablet d972dd11fcd9484f9a59f63d89f1e6e2 from peer 1b4b70e4a9d54c909b06663e702d4964 at {username='slave'} at 127.23.104.200:49621: session id = 1b4b70e4a9d54c909b06663e702d4964-d972dd11fcd9484f9a59f63d89f1e6e2
I20240503 02:12:54.293704 26419 tablet_copy_source_session.cc:205] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:54.298071 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d972dd11fcd9484f9a59f63d89f1e6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.306591 30114 tablet_copy_client.cc:783] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:54.306857 30114 tablet_copy_client.cc:647] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:54.310170 30114 tablet_copy_client.cc:523] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:54.313071 30114 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap starting.
I20240503 02:12:54.346298 30120 ts_tablet_manager.cc:902] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964: Initiating tablet copy from peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:12:54.348462 30120 tablet_copy_client.cc:308] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:39753
I20240503 02:12:54.353897 26159 tablet_copy_service.cc:140] P 2cf1cac003b44643a2d4de1204aacf0f: Received BeginTabletCopySession request for tablet c12cc2e0fb09431e95baae55803f2184 from peer 1b4b70e4a9d54c909b06663e702d4964 ({username='slave'} at 127.23.104.200:38329)
I20240503 02:12:54.354043 26159 tablet_copy_service.cc:161] P 2cf1cac003b44643a2d4de1204aacf0f: Beginning new tablet copy session on tablet c12cc2e0fb09431e95baae55803f2184 from peer 1b4b70e4a9d54c909b06663e702d4964 at {username='slave'} at 127.23.104.200:38329: session id = 1b4b70e4a9d54c909b06663e702d4964-c12cc2e0fb09431e95baae55803f2184
I20240503 02:12:54.357004 26159 tablet_copy_source_session.cc:205] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:54.358655 30120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12cc2e0fb09431e95baae55803f2184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.363747 30120 tablet_copy_client.cc:783] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:54.363997 30120 tablet_copy_client.cc:647] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:54.366276 30120 tablet_copy_client.cc:523] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:54.368577 30120 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap starting.
I20240503 02:12:54.384479 30114 log.cc:824] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:54.472672 30132 ts_tablet_manager.cc:902] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964: Initiating tablet copy from peer b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347)
I20240503 02:12:54.503516 30132 tablet_copy_client.cc:308] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.197:38347
I20240503 02:12:54.524343 30139 ts_tablet_manager.cc:902] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964: Initiating tablet copy from peer ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369)
I20240503 02:12:54.554116 26549 tablet_copy_service.cc:140] P b2835e59d40a48aea53c5dedb3bd1e9e: Received BeginTabletCopySession request for tablet 508ee4b913dd40b49d50be3fa4cdd6c4 from peer 1b4b70e4a9d54c909b06663e702d4964 ({username='slave'} at 127.23.104.200:44381)
I20240503 02:12:54.554272 26549 tablet_copy_service.cc:161] P b2835e59d40a48aea53c5dedb3bd1e9e: Beginning new tablet copy session on tablet 508ee4b913dd40b49d50be3fa4cdd6c4 from peer 1b4b70e4a9d54c909b06663e702d4964 at {username='slave'} at 127.23.104.200:44381: session id = 1b4b70e4a9d54c909b06663e702d4964-508ee4b913dd40b49d50be3fa4cdd6c4
I20240503 02:12:54.556968 26549 tablet_copy_source_session.cc:205] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:54.560835 30132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508ee4b913dd40b49d50be3fa4cdd6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.570680 30139 tablet_copy_client.cc:308] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:40369
I20240503 02:12:54.570770 30132 tablet_copy_client.cc:783] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:54.571012 30132 tablet_copy_client.cc:647] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:54.574503 30132 tablet_copy_client.cc:523] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:54.578265 30132 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap starting.
I20240503 02:12:54.592185 26679 tablet_copy_service.cc:140] P ae8b30bcfa5b425782a95374787dd230: Received BeginTabletCopySession request for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d from peer 1b4b70e4a9d54c909b06663e702d4964 ({username='slave'} at 127.23.104.200:53425)
I20240503 02:12:54.592345 26679 tablet_copy_service.cc:161] P ae8b30bcfa5b425782a95374787dd230: Beginning new tablet copy session on tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d from peer 1b4b70e4a9d54c909b06663e702d4964 at {username='slave'} at 127.23.104.200:53425: session id = 1b4b70e4a9d54c909b06663e702d4964-7eb1a7e7a1a440c1a826ac2f2e27db2d
I20240503 02:12:54.595052 26679 tablet_copy_source_session.cc:205] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:54.596917 30139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.621145 30139 tablet_copy_client.cc:783] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:54.621627 30139 tablet_copy_client.cc:647] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:54.628218 30151 ts_tablet_manager.cc:902] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964: Initiating tablet copy from peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143)
I20240503 02:12:54.628415 30139 tablet_copy_client.cc:523] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:54.637159 30139 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap starting.
I20240503 02:12:54.660905 30151 tablet_copy_client.cc:308] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.193:34143
I20240503 02:12:54.671192 26028 tablet_copy_service.cc:140] P 3bb103b2fd2344e7b55caf7389acf221: Received BeginTabletCopySession request for tablet 5332ecb1e4454eadb47cc721e2fbc487 from peer 1b4b70e4a9d54c909b06663e702d4964 ({username='slave'} at 127.23.104.200:42769)
I20240503 02:12:54.671347 26028 tablet_copy_service.cc:161] P 3bb103b2fd2344e7b55caf7389acf221: Beginning new tablet copy session on tablet 5332ecb1e4454eadb47cc721e2fbc487 from peer 1b4b70e4a9d54c909b06663e702d4964 at {username='slave'} at 127.23.104.200:42769: session id = 1b4b70e4a9d54c909b06663e702d4964-5332ecb1e4454eadb47cc721e2fbc487
I20240503 02:12:54.674275 26028 tablet_copy_source_session.cc:205] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:54.676307 30151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5332ecb1e4454eadb47cc721e2fbc487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.688382 30151 tablet_copy_client.cc:783] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:54.690675 30151 tablet_copy_client.cc:647] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:54.694072 30151 tablet_copy_client.cc:523] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:54.697417 30151 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap starting.
I20240503 02:12:54.726696 30151 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:54.727424 30151 tablet_bootstrap.cc:492] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap complete.
I20240503 02:12:54.728082 30151 ts_tablet_manager.cc:1367] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.000s
I20240503 02:12:54.734833 30151 raft_consensus.cc:348] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:54.735563 30151 raft_consensus.cc:729] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b4b70e4a9d54c909b06663e702d4964, State: Initialized, Role: LEARNER
I20240503 02:12:54.737546 30151 consensus_queue.cc:260] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:54.752308 30151 ts_tablet_manager.cc:1398] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964: Time spent starting tablet: real 0.024s user 0.006s sys 0.015s
I20240503 02:12:54.753191 26028 tablet_copy_service.cc:342] P 3bb103b2fd2344e7b55caf7389acf221: Request end of tablet copy session 1b4b70e4a9d54c909b06663e702d4964-5332ecb1e4454eadb47cc721e2fbc487 received from {username='slave'} at 127.23.104.200:42769
I20240503 02:12:54.753307 26028 tablet_copy_service.cc:433] P 3bb103b2fd2344e7b55caf7389acf221: ending tablet copy session 1b4b70e4a9d54c909b06663e702d4964-5332ecb1e4454eadb47cc721e2fbc487 on tablet 5332ecb1e4454eadb47cc721e2fbc487 with peer 1b4b70e4a9d54c909b06663e702d4964
I20240503 02:12:54.903988 30120 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap replayed 1/1 log segments. Stats: ops{read=267 overwritten=0 applied=267 ignored=0} inserts{seen=936 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:54.904500 30120 tablet_bootstrap.cc:492] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap complete.
I20240503 02:12:54.904832 30120 ts_tablet_manager.cc:1367] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964: Time spent bootstrapping tablet: real 0.536s user 0.306s sys 0.055s
I20240503 02:12:54.905800 30120 raft_consensus.cc:348] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:54.906116 30120 raft_consensus.cc:729] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b4b70e4a9d54c909b06663e702d4964, State: Initialized, Role: LEARNER
I20240503 02:12:54.906328 30120 consensus_queue.cc:260] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 267, Last appended: 1.267, Last appended by leader: 267, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:54.916157 30120 ts_tablet_manager.cc:1398] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240503 02:12:54.919960 26159 tablet_copy_service.cc:342] P 2cf1cac003b44643a2d4de1204aacf0f: Request end of tablet copy session 1b4b70e4a9d54c909b06663e702d4964-c12cc2e0fb09431e95baae55803f2184 received from {username='slave'} at 127.23.104.200:38329
I20240503 02:12:54.920075 26159 tablet_copy_service.cc:433] P 2cf1cac003b44643a2d4de1204aacf0f: ending tablet copy session 1b4b70e4a9d54c909b06663e702d4964-c12cc2e0fb09431e95baae55803f2184 on tablet c12cc2e0fb09431e95baae55803f2184 with peer 1b4b70e4a9d54c909b06663e702d4964
I20240503 02:12:55.037932 29880 raft_consensus.cc:1203] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:12:55.058784 30114 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap replayed 1/1 log segments. Stats: ops{read=523 overwritten=0 applied=523 ignored=0} inserts{seen=1205 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:55.059422 30114 tablet_bootstrap.cc:492] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap complete.
I20240503 02:12:55.059873 30114 ts_tablet_manager.cc:1367] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964: Time spent bootstrapping tablet: real 0.747s user 0.575s sys 0.061s
I20240503 02:12:55.060950 30114 raft_consensus.cc:348] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:55.062451 30114 raft_consensus.cc:729] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b4b70e4a9d54c909b06663e702d4964, State: Initialized, Role: LEARNER
I20240503 02:12:55.062696 30114 consensus_queue.cc:260] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 523, Last appended: 1.523, Last appended by leader: 523, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:55.063666 30114 ts_tablet_manager.cc:1398] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:55.064427 26419 tablet_copy_service.cc:342] P cff7b7a19fc64db5849391b1d8e10cce: Request end of tablet copy session 1b4b70e4a9d54c909b06663e702d4964-d972dd11fcd9484f9a59f63d89f1e6e2 received from {username='slave'} at 127.23.104.200:49621
I20240503 02:12:55.064565 26419 tablet_copy_service.cc:433] P cff7b7a19fc64db5849391b1d8e10cce: ending tablet copy session 1b4b70e4a9d54c909b06663e702d4964-d972dd11fcd9484f9a59f63d89f1e6e2 on tablet d972dd11fcd9484f9a59f63d89f1e6e2 with peer 1b4b70e4a9d54c909b06663e702d4964
I20240503 02:12:55.319623 29880 raft_consensus.cc:1203] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.522->[1.523-1.523] Dedup: 1.523->[]
I20240503 02:12:55.363530 30132 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap replayed 1/1 log segments. Stats: ops{read=520 overwritten=0 applied=520 ignored=0} inserts{seen=1202 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:55.364042 30132 tablet_bootstrap.cc:492] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap complete.
I20240503 02:12:55.364425 30132 ts_tablet_manager.cc:1367] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964: Time spent bootstrapping tablet: real 0.786s user 0.535s sys 0.076s
I20240503 02:12:55.365227 30132 raft_consensus.cc:348] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 520 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:55.365458 30132 raft_consensus.cc:729] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b4b70e4a9d54c909b06663e702d4964, State: Initialized, Role: LEARNER
I20240503 02:12:55.365655 30132 consensus_queue.cc:260] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 520, Last appended: 1.520, Last appended by leader: 520, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 520 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:55.366114 30132 ts_tablet_manager.cc:1398] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:55.366932 26549 tablet_copy_service.cc:342] P b2835e59d40a48aea53c5dedb3bd1e9e: Request end of tablet copy session 1b4b70e4a9d54c909b06663e702d4964-508ee4b913dd40b49d50be3fa4cdd6c4 received from {username='slave'} at 127.23.104.200:44381
I20240503 02:12:55.367053 26549 tablet_copy_service.cc:433] P b2835e59d40a48aea53c5dedb3bd1e9e: ending tablet copy session 1b4b70e4a9d54c909b06663e702d4964-508ee4b913dd40b49d50be3fa4cdd6c4 on tablet 508ee4b913dd40b49d50be3fa4cdd6c4 with peer 1b4b70e4a9d54c909b06663e702d4964
I20240503 02:12:55.382189 29880 raft_consensus.cc:1203] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.266->[1.267-1.267] Dedup: 1.267->[]
I20240503 02:12:55.463873 30139 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap replayed 1/1 log segments. Stats: ops{read=567 overwritten=0 applied=567 ignored=0} inserts{seen=1236 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:55.464335 30139 tablet_bootstrap.cc:492] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap complete.
I20240503 02:12:55.464686 30139 ts_tablet_manager.cc:1367] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964: Time spent bootstrapping tablet: real 0.828s user 0.556s sys 0.061s
I20240503 02:12:55.465368 30139 raft_consensus.cc:348] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 567 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:55.465579 30139 raft_consensus.cc:729] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b4b70e4a9d54c909b06663e702d4964, State: Initialized, Role: LEARNER
I20240503 02:12:55.465765 30139 consensus_queue.cc:260] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 567, Last appended: 1.567, Last appended by leader: 567, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 567 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:55.466274 30139 ts_tablet_manager.cc:1398] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:12:55.467002 26679 tablet_copy_service.cc:342] P ae8b30bcfa5b425782a95374787dd230: Request end of tablet copy session 1b4b70e4a9d54c909b06663e702d4964-7eb1a7e7a1a440c1a826ac2f2e27db2d received from {username='slave'} at 127.23.104.200:53425
I20240503 02:12:55.467113 26679 tablet_copy_service.cc:433] P ae8b30bcfa5b425782a95374787dd230: ending tablet copy session 1b4b70e4a9d54c909b06663e702d4964-7eb1a7e7a1a440c1a826ac2f2e27db2d on tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d with peer 1b4b70e4a9d54c909b06663e702d4964
I20240503 02:12:55.564940 30193 raft_consensus.cc:1050] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221: attempting to promote NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 to VOTER
I20240503 02:12:55.565747 30193 consensus_queue.cc:237] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:55.568219 29880 raft_consensus.cc:1261] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:55.568270 26524 raft_consensus.cc:1261] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:55.569860 27594 raft_consensus.cc:1261] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:55.570227 30232 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:55.570719 30234 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:55.571451 30236 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:55.571815 26264 raft_consensus.cc:1261] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:55.578024 30236 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:55.586755 26654 raft_consensus.cc:1261] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:55.587617 30234 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:55.588909 30236 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.589403 26264 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.594231 27594 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.597344 25858 catalog_manager.cc:5484] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 reported cstate change: config changed from index 2 to 3, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3bb103b2fd2344e7b55caf7389acf221" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:55.597268 29880 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.600646 26524 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.608971 26656 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } attrs { replace: true } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.622309 26007 consensus_queue.cc:237] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:55.624480 29880 raft_consensus.cc:1261] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:55.624724 26524 raft_consensus.cc:1261] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:55.624998 27594 raft_consensus.cc:1261] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:55.625252 30236 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:55.625664 26656 raft_consensus.cc:1261] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 3bb103b2fd2344e7b55caf7389acf221: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:55.625695 30237 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:55.627876 30236 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:55.628209 30234 consensus_queue.cc:1035] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:55.629276 30236 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.632591 26656 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.633638 25851 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5332ecb1e4454eadb47cc721e2fbc487 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:55.634537 27591 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.634537 29880 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.638710 25862 catalog_manager.cc:5484] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 reported cstate change: config changed from index 3 to 4, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "3bb103b2fd2344e7b55caf7389acf221" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.642454 26528 raft_consensus.cc:2932] T 5332ecb1e4454eadb47cc721e2fbc487 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7c384005cb3041a0bd70e163473b6acb (127.23.104.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.644605 26247 tablet_service.cc:1514] Processing DeleteTablet for tablet 5332ecb1e4454eadb47cc721e2fbc487 with delete_type TABLET_DATA_TOMBSTONED (TS 7c384005cb3041a0bd70e163473b6acb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37992
I20240503 02:12:55.645735 30259 tablet_replica.cc:330] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb: stopping tablet replica
I20240503 02:12:55.645977 30259 raft_consensus.cc:2229] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:55.646271 30259 raft_consensus.cc:2258] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.648487 30259 ts_tablet_manager.cc:1875] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:55.652647 30259 ts_tablet_manager.cc:1888] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:12:55.652828 30259 log.cc:1196] T 5332ecb1e4454eadb47cc721e2fbc487 P 7c384005cb3041a0bd70e163473b6acb: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/5332ecb1e4454eadb47cc721e2fbc487
I20240503 02:12:55.653663 25853 catalog_manager.cc:4830] TS 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999): tablet 5332ecb1e4454eadb47cc721e2fbc487 (table TestTable [id=0dc99a528f224b38bf5d65dde6674a5d]) successfully deleted
I20240503 02:12:55.865293 30275 raft_consensus.cc:1050] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce: attempting to promote NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 to VOTER
I20240503 02:12:55.866282 30275 consensus_queue.cc:237] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 523, Committed index: 523, Last appended: 1.523, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:55.869197 29880 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 523. Preceding OpId from leader: term: 1 index: 524. (index mismatch)
I20240503 02:12:55.869391 26656 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 523. Preceding OpId from leader: term: 1 index: 524. (index mismatch)
I20240503 02:12:55.869665 26007 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 523. Preceding OpId from leader: term: 1 index: 524. (index mismatch)
I20240503 02:12:55.869684 30274 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 524, Last known committed idx: 523, Time since last communication: 0.000s
I20240503 02:12:55.869871 26138 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 523. Preceding OpId from leader: term: 1 index: 524. (index mismatch)
I20240503 02:12:55.870383 27591 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 523. Preceding OpId from leader: term: 1 index: 524. (index mismatch)
I20240503 02:12:55.870548 30274 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 524, Last known committed idx: 523, Time since last communication: 0.000s
I20240503 02:12:55.871744 30027 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 524, Last known committed idx: 523, Time since last communication: 0.000s
I20240503 02:12:55.872005 30275 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 524, Last known committed idx: 523, Time since last communication: 0.000s
I20240503 02:12:55.872438 30027 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 524, Last known committed idx: 523, Time since last communication: 0.000s
I20240503 02:12:55.884641 30027 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 1 LEADER]: Committing config change with OpId 1.524: config changed from index 523 to 524, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.885381 26007 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.524: config changed from index 523 to 524, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.886098 27591 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.524: config changed from index 523 to 524, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.886343 26138 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.524: config changed from index 523 to 524, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.888983 29880 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.524: config changed from index 523 to 524, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.889652 26656 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.524: config changed from index 523 to 524, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.890759 25864 catalog_manager.cc:5484] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: config changed from index 523 to 524, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cff7b7a19fc64db5849391b1d8e10cce" committed_config { opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:55.897029 26393 consensus_queue.cc:237] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 524, Committed index: 524, Last appended: 1.524, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 525 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:55.899602 29880 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 524. Preceding OpId from leader: term: 1 index: 525. (index mismatch)
I20240503 02:12:55.900846 30277 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 525, Last known committed idx: 524, Time since last communication: 0.000s
I20240503 02:12:55.908432 27591 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 524. Preceding OpId from leader: term: 1 index: 525. (index mismatch)
I20240503 02:12:55.909005 30027 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 525, Last known committed idx: 524, Time since last communication: 0.000s
I20240503 02:12:55.911342 30275 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [term 1 LEADER]: Committing config change with OpId 1.525: config changed from index 524 to 525, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New config: { opid_index: 525 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.911916 27594 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.525: config changed from index 524 to 525, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New config: { opid_index: 525 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.916566 26008 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 524. Preceding OpId from leader: term: 1 index: 525. (index mismatch)
I20240503 02:12:55.916549 25853 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d972dd11fcd9484f9a59f63d89f1e6e2 with cas_config_opid_index 524: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:55.917690 30027 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 525, Last known committed idx: 524, Time since last communication: 0.000s
I20240503 02:12:55.918485 26138 raft_consensus.cc:1261] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 524. Preceding OpId from leader: term: 1 index: 525. (index mismatch)
I20240503 02:12:55.918731 29879 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.525: config changed from index 524 to 525, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New config: { opid_index: 525 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.921277 30027 consensus_queue.cc:1035] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 525, Last known committed idx: 524, Time since last communication: 0.000s
I20240503 02:12:55.922183 25863 catalog_manager.cc:5484] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: config changed from index 524 to 525, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New cstate: current_term: 1 leader_uuid: "cff7b7a19fc64db5849391b1d8e10cce" committed_config { opid_index: 525 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:55.922466 26007 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.525: config changed from index 524 to 525, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New config: { opid_index: 525 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.925913 26138 raft_consensus.cc:2932] T d972dd11fcd9484f9a59f63d89f1e6e2 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.525: config changed from index 524 to 525, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New config: { opid_index: 525 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.928608 26638 tablet_service.cc:1514] Processing DeleteTablet for tablet d972dd11fcd9484f9a59f63d89f1e6e2 with delete_type TABLET_DATA_TOMBSTONED (TS ae8b30bcfa5b425782a95374787dd230 not found in new config with opid_index 525) from {username='slave'} at 127.0.0.1:56314
I20240503 02:12:55.929237 30288 tablet_replica.cc:330] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230: stopping tablet replica
I20240503 02:12:55.929647 30288 raft_consensus.cc:2229] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:55.930130 30288 raft_consensus.cc:2258] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.934047 30288 ts_tablet_manager.cc:1875] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:55.937517 30286 raft_consensus.cc:1050] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f: attempting to promote NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 to VOTER
I20240503 02:12:55.938731 29879 raft_consensus.cc:1203] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.519->[1.520-1.520] Dedup: 1.520->[]
I20240503 02:12:55.940699 30286 consensus_queue.cc:237] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 1.267, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:55.944003 27594 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch)
I20240503 02:12:55.944465 29879 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch)
I20240503 02:12:55.945531 26528 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch)
I20240503 02:12:55.945537 26007 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch)
I20240503 02:12:55.945931 26393 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch)
I20240503 02:12:55.947419 30286 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.001s
I20240503 02:12:55.947675 30231 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20240503 02:12:55.947901 30289 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20240503 02:12:55.948540 30290 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20240503 02:12:55.949679 30288 ts_tablet_manager.cc:1888] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.524
I20240503 02:12:55.949823 30288 log.cc:1196] T d972dd11fcd9484f9a59f63d89f1e6e2 P ae8b30bcfa5b425782a95374787dd230: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal/wals/d972dd11fcd9484f9a59f63d89f1e6e2
I20240503 02:12:55.950770 25854 catalog_manager.cc:4830] TS ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369): tablet d972dd11fcd9484f9a59f63d89f1e6e2 (table rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]) successfully deleted
I20240503 02:12:55.966697 30290 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20240503 02:12:55.967418 30231 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Committing config change with OpId 1.268: config changed from index 267 to 268, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.972672 26007 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index 267 to 268, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.979420 26528 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index 267 to 268, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.985108 25859 catalog_manager.cc:5484] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 reported cstate change: config changed from index 267 to 268, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.985105 29879 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index 267 to 268, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.987571 27594 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index 267 to 268, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.989650 26393 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index 267 to 268, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { replace: true } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:55.998706 29879 raft_consensus.cc:1203] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.566->[1.567-1.567] Dedup: 1.567->[]
I20240503 02:12:56.052601 26264 consensus_queue.cc:237] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 514, Committed index: 514, Last appended: 1.514, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 515 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:56.055147 26656 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 514. Preceding OpId from leader: term: 1 index: 515. (index mismatch)
I20240503 02:12:56.055688 27594 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 514. Preceding OpId from leader: term: 1 index: 515. (index mismatch)
I20240503 02:12:56.056627 30175 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 515, Last known committed idx: 514, Time since last communication: 0.000s
I20240503 02:12:56.057201 30175 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 515, Last known committed idx: 514, Time since last communication: 0.000s
I20240503 02:12:56.067426 30175 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Committing config change with OpId 1.515: config changed from index 170 to 515, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 515 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.068689 26528 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 514. Preceding OpId from leader: term: 1 index: 515. (index mismatch)
I20240503 02:12:56.069635 26007 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 514. Preceding OpId from leader: term: 1 index: 515. (index mismatch)
I20240503 02:12:56.070165 30327 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 515, Last known committed idx: 514, Time since last communication: 0.000s
I20240503 02:12:56.070909 26656 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.515: config changed from index 170 to 515, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 515 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.071291 30328 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 515, Last known committed idx: 514, Time since last communication: 0.000s
I20240503 02:12:56.078804 25863 catalog_manager.cc:5484] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 reported cstate change: config changed from index 170 to 515, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: 515 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.080147 26007 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.515: config changed from index 170 to 515, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 515 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.080808 27594 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.515: config changed from index 170 to 515, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 515 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
W20240503 02:12:56.089574 26204 consensus_peers.cc:485] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb -> Peer 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857): Couldn't send request to peer 1b4b70e4a9d54c909b06663e702d4964. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21e7ddb56be1499b94920ebd252595ce. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:56.090137 26528 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.515: config changed from index 170 to 515, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 515 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.099138 26264 consensus_queue.cc:237] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 1.280, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:56.101316 26007 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 281. (index mismatch)
I20240503 02:12:56.101316 26528 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 281. (index mismatch)
I20240503 02:12:56.101871 30175 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20240503 02:12:56.102159 26656 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 281. (index mismatch)
I20240503 02:12:56.102648 27594 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 281. (index mismatch)
I20240503 02:12:56.102660 30175 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
W20240503 02:12:56.103107 26204 consensus_peers.cc:485] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb -> Peer 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857): Couldn't send request to peer 1b4b70e4a9d54c909b06663e702d4964. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c27502976d474b938cce2235e5784f79. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:56.103402 30327 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20240503 02:12:56.103667 30175 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20240503 02:12:56.106266 30327 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Committing config change with OpId 1.281: config changed from index 120 to 281, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.110040 26656 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.281: config changed from index 120 to 281, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.110504 26528 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.281: config changed from index 120 to 281, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.115716 26007 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.281: config changed from index 120 to 281, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.121129 25859 catalog_manager.cc:5484] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e reported cstate change: config changed from index 120 to 281, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.123757 27591 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.281: config changed from index 120 to 281, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.388710 30178 raft_consensus.cc:1050] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e: attempting to promote NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 to VOTER
I20240503 02:12:56.389868 30178 consensus_queue.cc:237] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 520, Committed index: 520, Last appended: 1.520, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 521 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:56.392828 26138 consensus_queue.cc:237] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 1.268, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:56.393462 26393 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 520. Preceding OpId from leader: term: 1 index: 521. (index mismatch)
I20240503 02:12:56.393640 27594 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 520. Preceding OpId from leader: term: 1 index: 521. (index mismatch)
I20240503 02:12:56.393975 30302 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 521, Last known committed idx: 520, Time since last communication: 0.000s
I20240503 02:12:56.394388 29879 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 520. Preceding OpId from leader: term: 1 index: 521. (index mismatch)
I20240503 02:12:56.394510 30302 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 521, Last known committed idx: 520, Time since last communication: 0.000s
I20240503 02:12:56.395478 26391 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
I20240503 02:12:56.395558 26528 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
I20240503 02:12:56.395632 29879 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
I20240503 02:12:56.396080 30290 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20240503 02:12:56.396318 30231 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20240503 02:12:56.396458 30302 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 521, Last known committed idx: 520, Time since last communication: 0.000s
I20240503 02:12:56.398484 27594 raft_consensus.cc:1261] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
I20240503 02:12:56.398898 30231 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20240503 02:12:56.403906 30349 raft_consensus.cc:1050] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230: attempting to promote NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 to VOTER
I20240503 02:12:56.404758 26656 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 520. Preceding OpId from leader: term: 1 index: 521. (index mismatch)
I20240503 02:12:56.404816 30349 consensus_queue.cc:237] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 567, Committed index: 567, Last appended: 1.567, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 568 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:56.405254 30378 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 521, Last known committed idx: 520, Time since last communication: 0.000s
I20240503 02:12:56.406961 26264 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 520. Preceding OpId from leader: term: 1 index: 521. (index mismatch)
I20240503 02:12:56.406813 30378 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 LEADER]: Committing config change with OpId 1.521: config changed from index 520 to 521, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 521 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.408012 29879 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.521: config changed from index 520 to 521, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 521 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.408555 30293 consensus_queue.cc:1035] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20240503 02:12:56.409415 30289 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Committing config change with OpId 1.269: config changed from index 268 to 269, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.410074 26528 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index 268 to 269, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.411499 29880 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index 268 to 269, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.413800 30302 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 521, Last known committed idx: 520, Time since last communication: 0.000s
I20240503 02:12:56.413808 27594 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index 268 to 269, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.415323 25851 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c12cc2e0fb09431e95baae55803f2184 with cas_config_opid_index 268: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:56.415920 25859 catalog_manager.cc:5484] T c12cc2e0fb09431e95baae55803f2184 P 2cf1cac003b44643a2d4de1204aacf0f reported cstate change: config changed from index 268 to 269, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:56.416525 26393 raft_consensus.cc:2932] T c12cc2e0fb09431e95baae55803f2184 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index 268 to 269, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.420272 25988 tablet_service.cc:1514] Processing DeleteTablet for tablet c12cc2e0fb09431e95baae55803f2184 with delete_type TABLET_DATA_TOMBSTONED (TS 3bb103b2fd2344e7b55caf7389acf221 not found in new config with opid_index 269) from {username='slave'} at 127.0.0.1:43226
I20240503 02:12:56.420207 25863 catalog_manager.cc:5484] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964 reported cstate change: config changed from index 520 to 521, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" committed_config { opid_index: 521 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.436873 26393 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.521: config changed from index 520 to 521, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 521 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.438395 30392 tablet_replica.cc:330] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221: stopping tablet replica
I20240503 02:12:56.439090 30392 raft_consensus.cc:2229] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.439868 30392 raft_consensus.cc:2258] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.439417 26264 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.521: config changed from index 520 to 521, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 521 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.440964 27594 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.521: config changed from index 520 to 521, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 521 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.441464 27591 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 567. Preceding OpId from leader: term: 1 index: 568. (index mismatch)
I20240503 02:12:56.441524 26007 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 567. Preceding OpId from leader: term: 1 index: 568. (index mismatch)
I20240503 02:12:56.442036 30069 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 568, Last known committed idx: 567, Time since last communication: 0.000s
I20240503 02:12:56.442638 30069 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 568, Last known committed idx: 567, Time since last communication: 0.000s
I20240503 02:12:56.444118 30392 ts_tablet_manager.cc:1875] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:56.447881 26656 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.521: config changed from index 520 to 521, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 521 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.454111 29880 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 567. Preceding OpId from leader: term: 1 index: 568. (index mismatch)
I20240503 02:12:56.455809 30069 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 568, Last known committed idx: 567, Time since last communication: 0.000s
I20240503 02:12:56.471455 26528 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 567. Preceding OpId from leader: term: 1 index: 568. (index mismatch)
I20240503 02:12:56.471940 30069 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 568, Last known committed idx: 567, Time since last communication: 0.000s
I20240503 02:12:56.472294 26138 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 567. Preceding OpId from leader: term: 1 index: 568. (index mismatch)
I20240503 02:12:56.473206 30069 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 568, Last known committed idx: 567, Time since last communication: 0.000s
I20240503 02:12:56.482743 30392 ts_tablet_manager.cc:1888] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.268
I20240503 02:12:56.483242 30392 log.cc:1196] T c12cc2e0fb09431e95baae55803f2184 P 3bb103b2fd2344e7b55caf7389acf221: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/c12cc2e0fb09431e95baae55803f2184
I20240503 02:12:56.482697 30069 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 1 LEADER]: Committing config change with OpId 1.568: config changed from index 567 to 568, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 568 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.484618 25851 catalog_manager.cc:4830] TS 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): tablet c12cc2e0fb09431e95baae55803f2184 (table rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]) successfully deleted
I20240503 02:12:56.484405 29879 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.568: config changed from index 567 to 568, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 568 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.488534 27591 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.568: config changed from index 567 to 568, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 568 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.489492 26138 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.568: config changed from index 567 to 568, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 568 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.491230 26007 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.568: config changed from index 567 to 568, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 568 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.491636 25859 catalog_manager.cc:5484] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 reported cstate change: config changed from index 567 to 568, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae8b30bcfa5b425782a95374787dd230" committed_config { opid_index: 568 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:56.498796 26528 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.568: config changed from index 567 to 568, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 568 OBSOLETE_local: false peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.506223 26656 consensus_queue.cc:237] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 568, Committed index: 568, Last appended: 1.568, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:56.510188 27594 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 568. Preceding OpId from leader: term: 1 index: 569. (index mismatch)
I20240503 02:12:56.511746 30069 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 569, Last known committed idx: 568, Time since last communication: 0.000s
I20240503 02:12:56.515692 26007 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 568. Preceding OpId from leader: term: 1 index: 569. (index mismatch)
I20240503 02:12:56.515869 29880 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 568. Preceding OpId from leader: term: 1 index: 569. (index mismatch)
I20240503 02:12:56.516317 30349 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 569, Last known committed idx: 568, Time since last communication: 0.000s
I20240503 02:12:56.516533 26528 raft_consensus.cc:1261] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 1 index: 568. Preceding OpId from leader: term: 1 index: 569. (index mismatch)
I20240503 02:12:56.517168 30401 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 569, Last known committed idx: 568, Time since last communication: 0.000s
I20240503 02:12:56.518384 30400 ts_tablet_manager.cc:902] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964: Initiating tablet copy from peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999)
I20240503 02:12:56.519066 30393 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [term 1 LEADER]: Committing config change with OpId 1.569: config changed from index 568 to 569, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.519879 30349 consensus_queue.cc:1035] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 569, Last known committed idx: 568, Time since last communication: 0.000s
I20240503 02:12:56.520730 29878 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.569: config changed from index 568 to 569, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.521566 26528 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.569: config changed from index 568 to 569, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.522610 26007 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.569: config changed from index 568 to 569, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.525152 25854 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d with cas_config_opid_index 568: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:56.526603 25859 catalog_manager.cc:5484] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 1b4b70e4a9d54c909b06663e702d4964 reported cstate change: config changed from index 568 to 569, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "ae8b30bcfa5b425782a95374787dd230" committed_config { opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.526481 27591 raft_consensus.cc:2932] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.569: config changed from index 568 to 569, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.533624 26117 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d with delete_type TABLET_DATA_TOMBSTONED (TS 2cf1cac003b44643a2d4de1204aacf0f not found in new config with opid_index 569) from {username='slave'} at 127.0.0.1:54956
I20240503 02:12:56.544502 30410 tablet_replica.cc:330] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f: stopping tablet replica
I20240503 02:12:56.544692 30410 raft_consensus.cc:2229] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.544945 30410 raft_consensus.cc:2258] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.548622 30410 ts_tablet_manager.cc:1875] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:56.552958 30400 tablet_copy_client.cc:308] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:45999
I20240503 02:12:56.553930 30410 ts_tablet_manager.cc:1888] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.568
I20240503 02:12:56.554126 30410 log.cc:1196] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P 2cf1cac003b44643a2d4de1204aacf0f: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/7eb1a7e7a1a440c1a826ac2f2e27db2d
I20240503 02:12:56.554989 25851 catalog_manager.cc:4830] TS 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d (table rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]) successfully deleted
I20240503 02:12:56.559300 26289 tablet_copy_service.cc:140] P 7c384005cb3041a0bd70e163473b6acb: Received BeginTabletCopySession request for tablet c27502976d474b938cce2235e5784f79 from peer 1b4b70e4a9d54c909b06663e702d4964 ({username='slave'} at 127.23.104.200:35129)
I20240503 02:12:56.559482 26289 tablet_copy_service.cc:161] P 7c384005cb3041a0bd70e163473b6acb: Beginning new tablet copy session on tablet c27502976d474b938cce2235e5784f79 from peer 1b4b70e4a9d54c909b06663e702d4964 at {username='slave'} at 127.23.104.200:35129: session id = 1b4b70e4a9d54c909b06663e702d4964-c27502976d474b938cce2235e5784f79
I20240503 02:12:56.561702 26289 tablet_copy_source_session.cc:205] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:56.562976 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c27502976d474b938cce2235e5784f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.568888 30400 tablet_copy_client.cc:783] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:56.569096 30400 tablet_copy_client.cc:647] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:56.571131 30400 tablet_copy_client.cc:523] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:56.573807 30400 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap starting.
I20240503 02:12:56.581501 30422 ts_tablet_manager.cc:902] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964: Initiating tablet copy from peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999)
I20240503 02:12:56.592025 30422 tablet_copy_client.cc:308] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:45999
I20240503 02:12:56.592624 26289 tablet_copy_service.cc:140] P 7c384005cb3041a0bd70e163473b6acb: Received BeginTabletCopySession request for tablet 21e7ddb56be1499b94920ebd252595ce from peer 1b4b70e4a9d54c909b06663e702d4964 ({username='slave'} at 127.23.104.200:35129)
I20240503 02:12:56.592759 26289 tablet_copy_service.cc:161] P 7c384005cb3041a0bd70e163473b6acb: Beginning new tablet copy session on tablet 21e7ddb56be1499b94920ebd252595ce from peer 1b4b70e4a9d54c909b06663e702d4964 at {username='slave'} at 127.23.104.200:35129: session id = 1b4b70e4a9d54c909b06663e702d4964-21e7ddb56be1499b94920ebd252595ce
I20240503 02:12:56.595676 26289 tablet_copy_source_session.cc:205] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:56.598111 30422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e7ddb56be1499b94920ebd252595ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.599507 26656 consensus_queue.cc:237] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 495, Committed index: 495, Last appended: 3.495, Last appended by leader: 234, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:56.607342 30422 tablet_copy_client.cc:783] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:56.607587 30422 tablet_copy_client.cc:647] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 1 WAL segments...
W20240503 02:12:56.608634 26594 consensus_peers.cc:485] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 -> Peer 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857): Couldn't send request to peer 1b4b70e4a9d54c909b06663e702d4964. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a864ac954e046eda8925f5daf4a0f1f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:56.608976 26138 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 495. Preceding OpId from leader: term: 3 index: 496. (index mismatch)
I20240503 02:12:56.609488 27591 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 495. Preceding OpId from leader: term: 3 index: 496. (index mismatch)
I20240503 02:12:56.610025 30401 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 496, Last known committed idx: 495, Time since last communication: 0.000s
I20240503 02:12:56.610208 26264 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 495. Preceding OpId from leader: term: 3 index: 496. (index mismatch)
I20240503 02:12:56.610633 26393 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 495. Preceding OpId from leader: term: 3 index: 496. (index mismatch)
I20240503 02:12:56.611191 30401 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 496, Last known committed idx: 495, Time since last communication: 0.000s
I20240503 02:12:56.611439 30393 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 496, Last known committed idx: 495, Time since last communication: 0.000s
I20240503 02:12:56.612717 30351 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 496, Last known committed idx: 495, Time since last communication: 0.000s
I20240503 02:12:56.615512 30401 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 LEADER]: Committing config change with OpId 3.496: config changed from index 254 to 496, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.617115 26264 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Committing config change with OpId 3.496: config changed from index 254 to 496, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.625145 25859 catalog_manager.cc:5484] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb reported cstate change: config changed from index 254 to 496, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New cstate: current_term: 3 leader_uuid: "ae8b30bcfa5b425782a95374787dd230" committed_config { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.628687 27591 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 3 FOLLOWER]: Committing config change with OpId 3.496: config changed from index 254 to 496, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.638268 26393 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Committing config change with OpId 3.496: config changed from index 254 to 496, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.642851 30422 tablet_copy_client.cc:523] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:56.643576 26136 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Committing config change with OpId 3.496: config changed from index 254 to 496, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.646529 30422 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap starting.
I20240503 02:12:56.686892 26393 consensus_queue.cc:237] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 1.247, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:56.689590 26007 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
W20240503 02:12:56.689826 26334 consensus_peers.cc:485] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce -> Peer 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857): Couldn't send request to peer 1b4b70e4a9d54c909b06663e702d4964. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39d3ba2cbbad40d6a232e534c65d2d74. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:56.690112 26136 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
I20240503 02:12:56.690130 27591 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
I20240503 02:12:56.690778 26264 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
I20240503 02:12:56.691017 30274 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240503 02:12:56.691522 30274 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240503 02:12:56.692075 30277 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240503 02:12:56.692492 30277 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240503 02:12:56.720253 30275 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 1 LEADER]: Committing config change with OpId 1.248: config changed from index 87 to 248, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.724797 26264 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 87 to 248, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.727952 25862 catalog_manager.cc:5484] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: config changed from index 87 to 248, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "cff7b7a19fc64db5849391b1d8e10cce" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:56.730098 27591 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 87 to 248, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.734674 26008 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 87 to 248, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.748229 26136 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 87 to 248, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:56.841044 26528 consensus_queue.cc:237] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 521, Committed index: 521, Last appended: 1.521, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 522 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:56.843508 26656 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 521. Preceding OpId from leader: term: 1 index: 522. (index mismatch)
I20240503 02:12:56.844269 30384 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 522, Last known committed idx: 521, Time since last communication: 0.000s
I20240503 02:12:56.845561 29878 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 521. Preceding OpId from leader: term: 1 index: 522. (index mismatch)
I20240503 02:12:56.846626 26268 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 521. Preceding OpId from leader: term: 1 index: 522. (index mismatch)
I20240503 02:12:56.849238 27591 raft_consensus.cc:1261] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer b2835e59d40a48aea53c5dedb3bd1e9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 521. Preceding OpId from leader: term: 1 index: 522. (index mismatch)
I20240503 02:12:56.850064 30302 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 522, Last known committed idx: 521, Time since last communication: 0.000s
I20240503 02:12:56.851529 30384 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 522, Last known committed idx: 521, Time since last communication: 0.001s
I20240503 02:12:56.852622 30378 consensus_queue.cc:1035] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 522, Last known committed idx: 521, Time since last communication: 0.000s
I20240503 02:12:56.859297 27591 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.522: config changed from index 521 to 522, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New config: { opid_index: 522 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.857162 30302 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 LEADER]: Committing config change with OpId 1.522: config changed from index 521 to 522, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New config: { opid_index: 522 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.862270 26268 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.522: config changed from index 521 to 522, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New config: { opid_index: 522 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.862355 29878 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.522: config changed from index 521 to 522, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New config: { opid_index: 522 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.868213 25862 catalog_manager.cc:5484] T 508ee4b913dd40b49d50be3fa4cdd6c4 P 7c384005cb3041a0bd70e163473b6acb reported cstate change: config changed from index 521 to 522, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" committed_config { opid_index: 522 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.874017 26378 tablet_service.cc:1514] Processing DeleteTablet for tablet 508ee4b913dd40b49d50be3fa4cdd6c4 with delete_type TABLET_DATA_TOMBSTONED (TS cff7b7a19fc64db5849391b1d8e10cce not found in new config with opid_index 522) from {username='slave'} at 127.0.0.1:48358
I20240503 02:12:56.887087 25854 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 508ee4b913dd40b49d50be3fa4cdd6c4 with cas_config_opid_index 521: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:56.889067 26656 raft_consensus.cc:2932] T 508ee4b913dd40b49d50be3fa4cdd6c4 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.522: config changed from index 521 to 522, VOTER cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196) evicted. New config: { opid_index: 522 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:56.913980 30475 tablet_replica.cc:330] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce: stopping tablet replica
I20240503 02:12:56.915932 30475 raft_consensus.cc:2229] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.916229 30475 raft_consensus.cc:2258] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.920583 30475 ts_tablet_manager.cc:1875] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:56.933777 30475 ts_tablet_manager.cc:1888] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.521
I20240503 02:12:56.935555 30475 log.cc:1196] T 508ee4b913dd40b49d50be3fa4cdd6c4 P cff7b7a19fc64db5849391b1d8e10cce: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/508ee4b913dd40b49d50be3fa4cdd6c4
I20240503 02:12:56.937849 25853 catalog_manager.cc:4830] TS cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675): tablet 508ee4b913dd40b49d50be3fa4cdd6c4 (table rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]) successfully deleted
I20240503 02:12:56.937984 30400 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap replayed 1/1 log segments. Stats: ops{read=281 overwritten=0 applied=281 ignored=0} inserts{seen=909 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:56.938457 30400 tablet_bootstrap.cc:492] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap complete.
I20240503 02:12:56.938745 30400 ts_tablet_manager.cc:1367] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964: Time spent bootstrapping tablet: real 0.365s user 0.286s sys 0.056s
I20240503 02:12:56.939424 30400 raft_consensus.cc:348] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:56.939639 30400 raft_consensus.cc:729] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b4b70e4a9d54c909b06663e702d4964, State: Initialized, Role: LEARNER
I20240503 02:12:56.939864 30400 consensus_queue.cc:260] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 281, Last appended: 1.281, Last appended by leader: 281, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:56.940270 30400 ts_tablet_manager.cc:1398] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 02:12:56.940821 26289 tablet_copy_service.cc:342] P 7c384005cb3041a0bd70e163473b6acb: Request end of tablet copy session 1b4b70e4a9d54c909b06663e702d4964-c27502976d474b938cce2235e5784f79 received from {username='slave'} at 127.23.104.200:35129
I20240503 02:12:56.940958 26289 tablet_copy_service.cc:433] P 7c384005cb3041a0bd70e163473b6acb: ending tablet copy session 1b4b70e4a9d54c909b06663e702d4964-c27502976d474b938cce2235e5784f79 on tablet c27502976d474b938cce2235e5784f79 with peer 1b4b70e4a9d54c909b06663e702d4964
I20240503 02:12:56.991546 29878 raft_consensus.cc:1203] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.280->[1.281-1.281] Dedup: 1.281->[]
I20240503 02:12:57.075770 30400 ts_tablet_manager.cc:902] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964: Initiating tablet copy from peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675)
I20240503 02:12:57.098806 30400 tablet_copy_client.cc:308] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:43675
I20240503 02:12:57.099498 26419 tablet_copy_service.cc:140] P cff7b7a19fc64db5849391b1d8e10cce: Received BeginTabletCopySession request for tablet 39d3ba2cbbad40d6a232e534c65d2d74 from peer 1b4b70e4a9d54c909b06663e702d4964 ({username='slave'} at 127.23.104.200:49621)
I20240503 02:12:57.099707 26419 tablet_copy_service.cc:161] P cff7b7a19fc64db5849391b1d8e10cce: Beginning new tablet copy session on tablet 39d3ba2cbbad40d6a232e534c65d2d74 from peer 1b4b70e4a9d54c909b06663e702d4964 at {username='slave'} at 127.23.104.200:49621: session id = 1b4b70e4a9d54c909b06663e702d4964-39d3ba2cbbad40d6a232e534c65d2d74
I20240503 02:12:57.102471 26419 tablet_copy_source_session.cc:205] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:57.104547 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d3ba2cbbad40d6a232e534c65d2d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.111013 30400 tablet_copy_client.cc:783] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:57.111258 30400 tablet_copy_client.cc:647] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:57.114260 30400 tablet_copy_client.cc:523] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:57.117782 30400 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap starting.
I20240503 02:12:57.140786 30509 ts_tablet_manager.cc:902] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964: Initiating tablet copy from peer ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369)
I20240503 02:12:57.147373 26136 consensus_queue.cc:237] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 530, Committed index: 530, Last appended: 1.530, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 531 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:57.151057 26528 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 530. Preceding OpId from leader: term: 1 index: 531. (index mismatch)
I20240503 02:12:57.151049 26656 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 530. Preceding OpId from leader: term: 1 index: 531. (index mismatch)
W20240503 02:12:57.151799 26074 consensus_peers.cc:485] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f -> Peer 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857): Couldn't send request to peer 1b4b70e4a9d54c909b06663e702d4964. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 041ffbedf2c04dbea4661b4c33f78ac1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:57.152102 30231 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 531, Last known committed idx: 530, Time since last communication: 0.000s
I20240503 02:12:57.152199 26007 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 530. Preceding OpId from leader: term: 1 index: 531. (index mismatch)
I20240503 02:12:57.152599 30231 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 531, Last known committed idx: 530, Time since last communication: 0.000s
I20240503 02:12:57.154922 30290 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 531, Last known committed idx: 530, Time since last communication: 0.000s
I20240503 02:12:57.165933 26393 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 530. Preceding OpId from leader: term: 1 index: 531. (index mismatch)
I20240503 02:12:57.166530 30231 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 531, Last known committed idx: 530, Time since last communication: 0.000s
I20240503 02:12:57.173086 30509 tablet_copy_client.cc:308] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:40369
I20240503 02:12:57.173750 26679 tablet_copy_service.cc:140] P ae8b30bcfa5b425782a95374787dd230: Received BeginTabletCopySession request for tablet 4a864ac954e046eda8925f5daf4a0f1f from peer 1b4b70e4a9d54c909b06663e702d4964 ({username='slave'} at 127.23.104.200:53425)
I20240503 02:12:57.173902 26679 tablet_copy_service.cc:161] P ae8b30bcfa5b425782a95374787dd230: Beginning new tablet copy session on tablet 4a864ac954e046eda8925f5daf4a0f1f from peer 1b4b70e4a9d54c909b06663e702d4964 at {username='slave'} at 127.23.104.200:53425: session id = 1b4b70e4a9d54c909b06663e702d4964-4a864ac954e046eda8925f5daf4a0f1f
I20240503 02:12:57.177820 26679 tablet_copy_source_session.cc:205] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:57.179350 30509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a864ac954e046eda8925f5daf4a0f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.193004 30231 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Committing config change with OpId 1.531: config changed from index 171 to 531, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 531 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:57.196321 30509 tablet_copy_client.cc:783] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:57.196717 30509 tablet_copy_client.cc:647] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:57.198210 26524 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.531: config changed from index 171 to 531, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 531 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:57.206883 25862 catalog_manager.cc:5484] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e reported cstate change: config changed from index 171 to 531, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: 531 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:57.211087 30509 tablet_copy_client.cc:523] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:57.215277 30509 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap starting.
I20240503 02:12:57.220399 26656 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.531: config changed from index 171 to 531, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 531 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:57.245540 30422 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap replayed 1/1 log segments. Stats: ops{read=515 overwritten=0 applied=515 ignored=0} inserts{seen=1143 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:57.245992 30422 tablet_bootstrap.cc:492] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap complete.
I20240503 02:12:57.246328 30422 ts_tablet_manager.cc:1367] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964: Time spent bootstrapping tablet: real 0.600s user 0.533s sys 0.048s
I20240503 02:12:57.247057 30422 raft_consensus.cc:348] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 515 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:57.247340 30422 raft_consensus.cc:729] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b4b70e4a9d54c909b06663e702d4964, State: Initialized, Role: LEARNER
I20240503 02:12:57.247538 30422 consensus_queue.cc:260] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 515, Last appended: 1.515, Last appended by leader: 515, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 515 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:57.248176 30422 ts_tablet_manager.cc:1398] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240503 02:12:57.248804 26289 tablet_copy_service.cc:342] P 7c384005cb3041a0bd70e163473b6acb: Request end of tablet copy session 1b4b70e4a9d54c909b06663e702d4964-21e7ddb56be1499b94920ebd252595ce received from {username='slave'} at 127.23.104.200:35129
I20240503 02:12:57.248929 26289 tablet_copy_service.cc:433] P 7c384005cb3041a0bd70e163473b6acb: ending tablet copy session 1b4b70e4a9d54c909b06663e702d4964-21e7ddb56be1499b94920ebd252595ce on tablet 21e7ddb56be1499b94920ebd252595ce with peer 1b4b70e4a9d54c909b06663e702d4964
I20240503 02:12:57.256996 26007 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.531: config changed from index 171 to 531, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 531 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:57.267583 26391 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.531: config changed from index 171 to 531, NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) added. New config: { opid_index: 531 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } } }
I20240503 02:12:57.482095 30327 raft_consensus.cc:1050] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb: attempting to promote NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 to VOTER
I20240503 02:12:57.483059 30327 consensus_queue.cc:237] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 1.281, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:57.488757 29878 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 1 index: 282. (index mismatch)
I20240503 02:12:57.489409 26528 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 1 index: 282. (index mismatch)
I20240503 02:12:57.490068 30333 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20240503 02:12:57.490556 30333 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20240503 02:12:57.494140 26007 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 1 index: 282. (index mismatch)
I20240503 02:12:57.494796 30175 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20240503 02:12:57.503185 27591 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 1 index: 282. (index mismatch)
I20240503 02:12:57.504635 30333 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20240503 02:12:57.515875 30400 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap replayed 1/1 log segments. Stats: ops{read=248 overwritten=0 applied=248 ignored=0} inserts{seen=930 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:57.516350 30400 tablet_bootstrap.cc:492] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap complete.
I20240503 02:12:57.516707 30400 ts_tablet_manager.cc:1367] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964: Time spent bootstrapping tablet: real 0.399s user 0.267s sys 0.057s
I20240503 02:12:57.517716 30400 raft_consensus.cc:348] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:57.518043 30400 raft_consensus.cc:729] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b4b70e4a9d54c909b06663e702d4964, State: Initialized, Role: LEARNER
I20240503 02:12:57.518337 30400 consensus_queue.cc:260] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 248, Last appended: 1.248, Last appended by leader: 248, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:57.519956 30400 ts_tablet_manager.cc:1398] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:57.520972 26656 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 1 index: 282. (index mismatch)
I20240503 02:12:57.521839 26419 tablet_copy_service.cc:342] P cff7b7a19fc64db5849391b1d8e10cce: Request end of tablet copy session 1b4b70e4a9d54c909b06663e702d4964-39d3ba2cbbad40d6a232e534c65d2d74 received from {username='slave'} at 127.23.104.200:49621
I20240503 02:12:57.521965 26419 tablet_copy_service.cc:433] P cff7b7a19fc64db5849391b1d8e10cce: ending tablet copy session 1b4b70e4a9d54c909b06663e702d4964-39d3ba2cbbad40d6a232e534c65d2d74 on tablet 39d3ba2cbbad40d6a232e534c65d2d74 with peer 1b4b70e4a9d54c909b06663e702d4964
I20240503 02:12:57.522367 30327 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20240503 02:12:57.526335 30327 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Committing config change with OpId 1.282: config changed from index 281 to 282, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:57.527031 26528 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.282: config changed from index 281 to 282, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:57.527364 29880 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.282: config changed from index 281 to 282, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:57.532470 25859 catalog_manager.cc:5484] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 reported cstate change: config changed from index 281 to 282, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:57.535238 26007 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.282: config changed from index 281 to 282, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:57.535264 26656 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.282: config changed from index 281 to 282, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:57.539969 27591 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.282: config changed from index 281 to 282, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } attrs { replace: true } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:57.553788 26268 consensus_queue.cc:237] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 1.282, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:57.556174 29880 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20240503 02:12:57.556447 26007 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20240503 02:12:57.556694 30175 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20240503 02:12:57.557138 27591 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20240503 02:12:57.557471 30175 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20240503 02:12:57.557943 30327 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20240503 02:12:57.559510 26656 raft_consensus.cc:1261] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20240503 02:12:57.560029 30559 consensus_queue.cc:1035] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20240503 02:12:57.560743 30559 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Committing config change with OpId 1.283: config changed from index 282 to 283, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:57.561188 26007 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 282 to 283, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:57.563196 26656 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 282 to 283, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:57.565048 25853 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c27502976d474b938cce2235e5784f79 with cas_config_opid_index 282: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:57.566668 25863 catalog_manager.cc:5484] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb reported cstate change: config changed from index 282 to 283, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:57.568116 29880 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 282 to 283, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:57.570199 26509 tablet_service.cc:1514] Processing DeleteTablet for tablet c27502976d474b938cce2235e5784f79 with delete_type TABLET_DATA_TOMBSTONED (TS b2835e59d40a48aea53c5dedb3bd1e9e not found in new config with opid_index 283) from {username='slave'} at 127.0.0.1:34002
I20240503 02:12:57.570828 30561 tablet_replica.cc:330] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e: stopping tablet replica
I20240503 02:12:57.572974 27591 raft_consensus.cc:2932] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 282 to 283, VOTER b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:57.578239 30561 raft_consensus.cc:2229] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.578605 30561 raft_consensus.cc:2258] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.583115 30561 ts_tablet_manager.cc:1875] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:57.592176 30561 ts_tablet_manager.cc:1888] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.282
I20240503 02:12:57.592288 30561 log.cc:1196] T c27502976d474b938cce2235e5784f79 P b2835e59d40a48aea53c5dedb3bd1e9e: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/wals/c27502976d474b938cce2235e5784f79
I20240503 02:12:57.593017 25854 catalog_manager.cc:4830] TS b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347): tablet c27502976d474b938cce2235e5784f79 (table rebalance_test_table_3 [id=02e35d1522244be29c80c4b1a91bff24]) successfully deleted
I20240503 02:12:57.665642 29880 raft_consensus.cc:1203] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.514->[1.515-1.515] Dedup: 1.515->[]
I20240503 02:12:57.697479 30400 ts_tablet_manager.cc:902] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964: Initiating tablet copy from peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:12:57.698752 30400 tablet_copy_client.cc:308] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:39753
I20240503 02:12:57.699481 26159 tablet_copy_service.cc:140] P 2cf1cac003b44643a2d4de1204aacf0f: Received BeginTabletCopySession request for tablet 041ffbedf2c04dbea4661b4c33f78ac1 from peer 1b4b70e4a9d54c909b06663e702d4964 ({username='slave'} at 127.23.104.200:38329)
I20240503 02:12:57.699644 26159 tablet_copy_service.cc:161] P 2cf1cac003b44643a2d4de1204aacf0f: Beginning new tablet copy session on tablet 041ffbedf2c04dbea4661b4c33f78ac1 from peer 1b4b70e4a9d54c909b06663e702d4964 at {username='slave'} at 127.23.104.200:38329: session id = 1b4b70e4a9d54c909b06663e702d4964-041ffbedf2c04dbea4661b4c33f78ac1
I20240503 02:12:57.702466 26159 tablet_copy_source_session.cc:205] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:57.703860 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041ffbedf2c04dbea4661b4c33f78ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.710635 30400 tablet_copy_client.cc:783] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:57.710881 30400 tablet_copy_client.cc:647] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:57.714555 30400 tablet_copy_client.cc:523] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:57.717912 30400 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap starting.
I20240503 02:12:57.827632 30509 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap replayed 1/1 log segments. Stats: ops{read=496 overwritten=0 applied=496 ignored=0} inserts{seen=1123 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:57.828111 30509 tablet_bootstrap.cc:492] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap complete.
I20240503 02:12:57.828450 30509 ts_tablet_manager.cc:1367] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964: Time spent bootstrapping tablet: real 0.613s user 0.490s sys 0.049s
I20240503 02:12:57.829277 30509 raft_consensus.cc:348] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:57.829561 30509 raft_consensus.cc:729] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 1b4b70e4a9d54c909b06663e702d4964, State: Initialized, Role: LEARNER
I20240503 02:12:57.829769 30509 consensus_queue.cc:260] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 496, Last appended: 3.496, Last appended by leader: 496, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:57.830387 30509 ts_tablet_manager.cc:1398] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:57.831907 26679 tablet_copy_service.cc:342] P ae8b30bcfa5b425782a95374787dd230: Request end of tablet copy session 1b4b70e4a9d54c909b06663e702d4964-4a864ac954e046eda8925f5daf4a0f1f received from {username='slave'} at 127.23.104.200:53425
I20240503 02:12:57.832026 26679 tablet_copy_service.cc:433] P ae8b30bcfa5b425782a95374787dd230: ending tablet copy session 1b4b70e4a9d54c909b06663e702d4964-4a864ac954e046eda8925f5daf4a0f1f on tablet 4a864ac954e046eda8925f5daf4a0f1f with peer 1b4b70e4a9d54c909b06663e702d4964
I20240503 02:12:57.998827 29878 raft_consensus.cc:1203] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.495->[3.496-3.496] Dedup: 3.496->[]
I20240503 02:12:58.053210 29878 raft_consensus.cc:1203] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.247->[1.248-1.248] Dedup: 1.248->[]
I20240503 02:12:58.059147 30556 raft_consensus.cc:1050] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb: attempting to promote NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 to VOTER
I20240503 02:12:58.060420 30556 consensus_queue.cc:237] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 515, Committed index: 515, Last appended: 1.515, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:58.069447 27591 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 515. Preceding OpId from leader: term: 1 index: 516. (index mismatch)
I20240503 02:12:58.070449 30175 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 516, Last known committed idx: 515, Time since last communication: 0.000s
I20240503 02:12:58.074829 26656 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 515. Preceding OpId from leader: term: 1 index: 516. (index mismatch)
I20240503 02:12:58.075451 30175 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 516, Last known committed idx: 515, Time since last communication: 0.000s
I20240503 02:12:58.080242 29878 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 515. Preceding OpId from leader: term: 1 index: 516. (index mismatch)
I20240503 02:12:58.080798 30175 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 516, Last known committed idx: 515, Time since last communication: 0.000s
I20240503 02:12:58.081952 26008 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 515. Preceding OpId from leader: term: 1 index: 516. (index mismatch)
I20240503 02:12:58.082144 26528 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 515. Preceding OpId from leader: term: 1 index: 516. (index mismatch)
I20240503 02:12:58.082754 30327 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 516, Last known committed idx: 515, Time since last communication: 0.000s
I20240503 02:12:58.084311 30327 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 516, Last known committed idx: 515, Time since last communication: 0.000s
I20240503 02:12:58.131263 30327 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Committing config change with OpId 1.516: config changed from index 515 to 516, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.137560 29880 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.516: config changed from index 515 to 516, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.145850 25863 catalog_manager.cc:5484] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 reported cstate change: config changed from index 515 to 516, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.150573 27591 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.516: config changed from index 515 to 516, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.183120 26656 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.516: config changed from index 515 to 516, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.192153 26528 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.516: config changed from index 515 to 516, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.205788 26008 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.516: config changed from index 515 to 516, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.233397 26268 consensus_queue.cc:237] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 516, Committed index: 516, Last appended: 1.516, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:58.236157 26008 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 516. Preceding OpId from leader: term: 1 index: 517. (index mismatch)
I20240503 02:12:58.236702 29880 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 516. Preceding OpId from leader: term: 1 index: 517. (index mismatch)
I20240503 02:12:58.236733 30175 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 517, Last known committed idx: 516, Time since last communication: 0.000s
I20240503 02:12:58.238734 26656 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 516. Preceding OpId from leader: term: 1 index: 517. (index mismatch)
I20240503 02:12:58.238866 26528 raft_consensus.cc:1261] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 7c384005cb3041a0bd70e163473b6acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 516. Preceding OpId from leader: term: 1 index: 517. (index mismatch)
I20240503 02:12:58.239449 30327 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 517, Last known committed idx: 516, Time since last communication: 0.000s
I20240503 02:12:58.239912 30327 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 517, Last known committed idx: 516, Time since last communication: 0.000s
I20240503 02:12:58.242803 30175 consensus_queue.cc:1035] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 517, Last known committed idx: 516, Time since last communication: 0.000s
I20240503 02:12:58.245199 30327 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb [term 1 LEADER]: Committing config change with OpId 1.517: config changed from index 516 to 517, VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) evicted. New config: { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.246943 26007 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.517: config changed from index 516 to 517, VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) evicted. New config: { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.250960 26528 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.517: config changed from index 516 to 517, VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) evicted. New config: { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.253396 26654 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.517: config changed from index 516 to 517, VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) evicted. New config: { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.254165 25859 catalog_manager.cc:5484] T 21e7ddb56be1499b94920ebd252595ce P 3bb103b2fd2344e7b55caf7389acf221 reported cstate change: config changed from index 516 to 517, VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) evicted. New cstate: current_term: 1 leader_uuid: "7c384005cb3041a0bd70e163473b6acb" committed_config { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.259953 25853 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21e7ddb56be1499b94920ebd252595ce with cas_config_opid_index 516: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:58.261162 29880 raft_consensus.cc:2932] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.517: config changed from index 516 to 517, VOTER 0a801f0897884f108cdb89434b18628a (127.23.104.199) evicted. New config: { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.297155 27573 tablet_service.cc:1514] Processing DeleteTablet for tablet 21e7ddb56be1499b94920ebd252595ce with delete_type TABLET_DATA_TOMBSTONED (TS 0a801f0897884f108cdb89434b18628a not found in new config with opid_index 517) from {username='slave'} at 127.0.0.1:51772
I20240503 02:12:58.298933 30657 tablet_replica.cc:330] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a: stopping tablet replica
I20240503 02:12:58.299454 30657 raft_consensus.cc:2229] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.300073 30657 raft_consensus.cc:2258] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.302600 30657 ts_tablet_manager.cc:1875] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:58.308830 30657 ts_tablet_manager.cc:1888] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.516
I20240503 02:12:58.308965 30657 log.cc:1196] T 21e7ddb56be1499b94920ebd252595ce P 0a801f0897884f108cdb89434b18628a: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal/wals/21e7ddb56be1499b94920ebd252595ce
I20240503 02:12:58.309801 25851 catalog_manager.cc:4830] TS 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): tablet 21e7ddb56be1499b94920ebd252595ce (table rebalance_test_table_4 [id=6e0193858acc4526bc2d4a6f4918b283]) successfully deleted
I20240503 02:12:58.375486 30400 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap replayed 1/1 log segments. Stats: ops{read=531 overwritten=0 applied=531 ignored=0} inserts{seen=1159 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:58.375876 30400 tablet_bootstrap.cc:492] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964: Bootstrap complete.
I20240503 02:12:58.376145 30400 ts_tablet_manager.cc:1367] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964: Time spent bootstrapping tablet: real 0.658s user 0.534s sys 0.082s
I20240503 02:12:58.376686 30400 raft_consensus.cc:348] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 531 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:58.376883 30400 raft_consensus.cc:729] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b4b70e4a9d54c909b06663e702d4964, State: Initialized, Role: LEARNER
I20240503 02:12:58.377048 30400 consensus_queue.cc:260] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 531, Last appended: 1.531, Last appended by leader: 531, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 531 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: true } }
I20240503 02:12:58.377607 30400 ts_tablet_manager.cc:1398] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:12:58.378224 26159 tablet_copy_service.cc:342] P 2cf1cac003b44643a2d4de1204aacf0f: Request end of tablet copy session 1b4b70e4a9d54c909b06663e702d4964-041ffbedf2c04dbea4661b4c33f78ac1 received from {username='slave'} at 127.23.104.200:38329
I20240503 02:12:58.378338 26159 tablet_copy_service.cc:433] P 2cf1cac003b44643a2d4de1204aacf0f: ending tablet copy session 1b4b70e4a9d54c909b06663e702d4964-041ffbedf2c04dbea4661b4c33f78ac1 on tablet 041ffbedf2c04dbea4661b4c33f78ac1 with peer 1b4b70e4a9d54c909b06663e702d4964
I20240503 02:12:58.455281 30562 raft_consensus.cc:1050] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230: attempting to promote NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 to VOTER
I20240503 02:12:58.456436 30562 consensus_queue.cc:237] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 496, Committed index: 496, Last appended: 3.496, Last appended by leader: 234, Current term: 3, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:58.458920 29880 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964 [term 3 LEARNER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 496. Preceding OpId from leader: term: 3 index: 497. (index mismatch)
I20240503 02:12:58.459066 26136 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 496. Preceding OpId from leader: term: 3 index: 497. (index mismatch)
I20240503 02:12:58.459237 27591 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 496. Preceding OpId from leader: term: 3 index: 497. (index mismatch)
I20240503 02:12:58.459470 30351 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I20240503 02:12:58.460244 30563 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I20240503 02:12:58.460510 30351 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I20240503 02:12:58.464834 30351 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 LEADER]: Committing config change with OpId 3.497: config changed from index 496 to 497, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.466138 29880 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964 [term 3 FOLLOWER]: Committing config change with OpId 3.497: config changed from index 496 to 497, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.469029 26268 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 496. Preceding OpId from leader: term: 3 index: 497. (index mismatch)
I20240503 02:12:58.469292 26391 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Refusing update from remote peer ae8b30bcfa5b425782a95374787dd230: Log matching property violated. Preceding OpId in replica: term: 3 index: 496. Preceding OpId from leader: term: 3 index: 497. (index mismatch)
I20240503 02:12:58.469069 26136 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Committing config change with OpId 3.497: config changed from index 496 to 497, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.469590 30563 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I20240503 02:12:58.470307 30563 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I20240503 02:12:58.473548 25866 catalog_manager.cc:5484] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964 reported cstate change: config changed from index 496 to 497, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "ae8b30bcfa5b425782a95374787dd230" committed_config { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.473670 27591 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 3 FOLLOWER]: Committing config change with OpId 3.497: config changed from index 496 to 497, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.474256 26268 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Committing config change with OpId 3.497: config changed from index 496 to 497, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.478564 26391 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Committing config change with OpId 3.497: config changed from index 496 to 497, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.557060 30636 raft_consensus.cc:1050] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce: attempting to promote NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 to VOTER
I20240503 02:12:58.558090 30636 consensus_queue.cc:237] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 1.248, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:58.560493 26007 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20240503 02:12:58.560703 26136 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20240503 02:12:58.560734 27591 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20240503 02:12:58.561409 30635 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20240503 02:12:58.561677 30550 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20240503 02:12:58.562019 30634 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20240503 02:12:58.562522 29880 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20240503 02:12:58.563652 30634 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20240503 02:12:58.577235 26268 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20240503 02:12:58.578488 30550 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20240503 02:12:58.579041 26007 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index 248 to 249, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.580390 30636 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 1 LEADER]: Committing config change with OpId 1.249: config changed from index 248 to 249, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.580829 29880 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index 248 to 249, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.584897 25859 catalog_manager.cc:5484] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 reported cstate change: config changed from index 248 to 249, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cff7b7a19fc64db5849391b1d8e10cce" committed_config { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.592017 26268 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index 248 to 249, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.593187 26136 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index 248 to 249, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.593391 27591 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index 248 to 249, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { replace: true } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:58.598536 29880 raft_consensus.cc:1203] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.530->[1.531-1.531] Dedup: 1.531->[]
I20240503 02:12:58.629734 26654 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f"
dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51206
I20240503 02:12:58.629981 26654 raft_consensus.cc:595] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 LEADER]: Received request to transfer leadership
I20240503 02:12:58.876302 26654 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f"
dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51230
I20240503 02:12:58.876470 26654 raft_consensus.cc:595] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 LEADER]: Received request to transfer leadership
I20240503 02:12:58.891459 30562 raft_consensus.cc:979] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230: : Instructing follower 0a801f0897884f108cdb89434b18628a to start an election
I20240503 02:12:58.891615 30562 raft_consensus.cc:1067] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 LEADER]: Signalling peer 0a801f0897884f108cdb89434b18628a to start an election
I20240503 02:12:58.892490 27594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f"
dest_uuid: "0a801f0897884f108cdb89434b18628a"
from {username='slave'} at 127.23.104.198:47523
I20240503 02:12:58.892769 27594 raft_consensus.cc:482] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:58.892946 27594 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:12:58.894955 27594 raft_consensus.cc:504] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:58.896975 26391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "0a801f0897884f108cdb89434b18628a" candidate_term: 4 candidate_status { last_received { term: 3 index: 497 } } ignore_live_leader: true dest_uuid: "cff7b7a19fc64db5849391b1d8e10cce"
I20240503 02:12:58.897187 26391 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:12:58.897120 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "0a801f0897884f108cdb89434b18628a" candidate_term: 4 candidate_status { last_received { term: 3 index: 497 } } ignore_live_leader: true dest_uuid: "7c384005cb3041a0bd70e163473b6acb"
I20240503 02:12:58.897282 26654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "0a801f0897884f108cdb89434b18628a" candidate_term: 4 candidate_status { last_received { term: 3 index: 497 } } ignore_live_leader: true dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
I20240503 02:12:58.897342 26268 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:12:58.897472 26654 raft_consensus.cc:3032] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:12:58.897545 26654 raft_consensus.cc:729] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae8b30bcfa5b425782a95374787dd230, State: Running, Role: LEADER
I20240503 02:12:58.897856 26654 consensus_queue.cc:260] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 497, Committed index: 497, Last appended: 3.497, Last appended by leader: 497, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:58.898172 26136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "0a801f0897884f108cdb89434b18628a" candidate_term: 4 candidate_status { last_received { term: 3 index: 497 } } ignore_live_leader: true dest_uuid: "2cf1cac003b44643a2d4de1204aacf0f"
I20240503 02:12:58.898377 26136 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:12:58.898411 26654 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:12:58.900209 26268 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a801f0897884f108cdb89434b18628a in term 4.
I20240503 02:12:58.901140 26654 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a801f0897884f108cdb89434b18628a in term 4.
I20240503 02:12:58.901732 27594 leader_election.cc:288] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 4 election: Requested vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857)
I20240503 02:12:58.903874 26391 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a801f0897884f108cdb89434b18628a in term 4.
I20240503 02:12:58.904732 27526 leader_election.cc:302] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 6 voters: 4 yes votes; 0 no votes. yes voters: 0a801f0897884f108cdb89434b18628a, 7c384005cb3041a0bd70e163473b6acb, ae8b30bcfa5b425782a95374787dd230, cff7b7a19fc64db5849391b1d8e10cce; no voters:
I20240503 02:12:58.904852 26136 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a801f0897884f108cdb89434b18628a in term 4.
I20240503 02:12:58.905633 30645 raft_consensus.cc:2781] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:12:58.906487 30645 raft_consensus.cc:686] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 4 LEADER]: Becoming Leader. State: Replica: 0a801f0897884f108cdb89434b18628a, State: Running, Role: LEADER
I20240503 02:12:58.906956 30645 consensus_queue.cc:237] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 497, Committed index: 497, Last appended: 3.497, Last appended by leader: 497, Current term: 4, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:58.911338 25864 catalog_manager.cc:5484] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a reported cstate change: term changed from 3 to 4, leader changed from ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) to 0a801f0897884f108cdb89434b18628a (127.23.104.199). New cstate: current_term: 4 leader_uuid: "0a801f0897884f108cdb89434b18628a" committed_config { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:58.915153 29880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a864ac954e046eda8925f5daf4a0f1f" candidate_uuid: "0a801f0897884f108cdb89434b18628a" candidate_term: 4 candidate_status { last_received { term: 3 index: 497 } } ignore_live_leader: true dest_uuid: "1b4b70e4a9d54c909b06663e702d4964"
I20240503 02:12:58.915470 29880 raft_consensus.cc:3037] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:12:58.917858 29880 raft_consensus.cc:2454] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a801f0897884f108cdb89434b18628a in term 4.
I20240503 02:12:59.041105 30524 raft_consensus.cc:1050] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f: attempting to promote NON_VOTER 1b4b70e4a9d54c909b06663e702d4964 to VOTER
I20240503 02:12:59.041832 30524 consensus_queue.cc:237] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 531, Committed index: 531, Last appended: 1.531, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:59.044332 26007 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 531. Preceding OpId from leader: term: 1 index: 532. (index mismatch)
I20240503 02:12:59.044732 26528 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 531. Preceding OpId from leader: term: 1 index: 532. (index mismatch)
I20240503 02:12:59.045204 30746 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 532, Last known committed idx: 531, Time since last communication: 0.000s
I20240503 02:12:59.045325 26391 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 531. Preceding OpId from leader: term: 1 index: 532. (index mismatch)
I20240503 02:12:59.045593 26654 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 531. Preceding OpId from leader: term: 1 index: 532. (index mismatch)
I20240503 02:12:59.045823 30706 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 532, Last known committed idx: 531, Time since last communication: 0.000s
I20240503 02:12:59.046082 30746 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 532, Last known committed idx: 531, Time since last communication: 0.000s
I20240503 02:12:59.046355 30290 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 532, Last known committed idx: 531, Time since last communication: 0.000s
I20240503 02:12:59.046483 29880 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 LEARNER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 531. Preceding OpId from leader: term: 1 index: 532. (index mismatch)
I20240503 02:12:59.047149 30706 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 532, Last known committed idx: 531, Time since last communication: 0.000s
I20240503 02:12:59.049978 30706 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Committing config change with OpId 1.532: config changed from index 531 to 532, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.050668 29880 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.532: config changed from index 531 to 532, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.051077 26391 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.532: config changed from index 531 to 532, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.054118 26528 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.532: config changed from index 531 to 532, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.054755 26007 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.532: config changed from index 531 to 532, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.057190 25864 catalog_manager.cc:5484] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f reported cstate change: config changed from index 531 to 532, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:59.057614 26654 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.532: config changed from index 531 to 532, 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } attrs { promote: false replace: true } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.062376 26391 consensus_queue.cc:237] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 249, Committed index: 249, Last appended: 1.249, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:59.064878 26136 consensus_queue.cc:237] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 532, Committed index: 532, Last appended: 1.532, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:59.065771 26007 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 249. Preceding OpId from leader: term: 1 index: 250. (index mismatch)
I20240503 02:12:59.065752 27594 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 249. Preceding OpId from leader: term: 1 index: 250. (index mismatch)
I20240503 02:12:59.065788 26268 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 249. Preceding OpId from leader: term: 1 index: 250. (index mismatch)
I20240503 02:12:59.066238 30634 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20240503 02:12:59.066694 29880 raft_consensus.cc:1261] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Refusing update from remote peer cff7b7a19fc64db5849391b1d8e10cce: Log matching property violated. Preceding OpId in replica: term: 1 index: 249. Preceding OpId from leader: term: 1 index: 250. (index mismatch)
I20240503 02:12:59.067297 30634 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20240503 02:12:59.067647 26654 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 532. Preceding OpId from leader: term: 1 index: 533. (index mismatch)
I20240503 02:12:59.067804 30550 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20240503 02:12:59.068189 30290 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 533, Last known committed idx: 532, Time since last communication: 0.000s
I20240503 02:12:59.068264 30695 consensus_queue.cc:1035] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20240503 02:12:59.068663 26528 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 532. Preceding OpId from leader: term: 1 index: 533. (index mismatch)
I20240503 02:12:59.069047 30290 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 533, Last known committed idx: 532, Time since last communication: 0.000s
I20240503 02:12:59.070999 29879 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 532. Preceding OpId from leader: term: 1 index: 533. (index mismatch)
I20240503 02:12:59.071611 26391 raft_consensus.cc:1261] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Refusing update from remote peer 2cf1cac003b44643a2d4de1204aacf0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 532. Preceding OpId from leader: term: 1 index: 533. (index mismatch)
I20240503 02:12:59.071614 30524 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 533, Last known committed idx: 532, Time since last communication: 0.000s
I20240503 02:12:59.071435 30290 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 LEADER]: Committing config change with OpId 1.533: config changed from index 532 to 533, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.072476 30524 consensus_queue.cc:1035] T 041ffbedf2c04dbea4661b4c33f78ac1 P 2cf1cac003b44643a2d4de1204aacf0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 533, Last known committed idx: 532, Time since last communication: 0.000s
I20240503 02:12:59.072505 26528 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e [term 1 FOLLOWER]: Committing config change with OpId 1.533: config changed from index 532 to 533, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.072549 30695 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce [term 1 LEADER]: Committing config change with OpId 1.250: config changed from index 249 to 250, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.074205 29878 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.533: config changed from index 532 to 533, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.076392 25853 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39d3ba2cbbad40d6a232e534c65d2d74 with cas_config_opid_index 249: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:59.076417 26656 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Committing config change with OpId 1.533: config changed from index 532 to 533, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.077634 25859 catalog_manager.cc:5484] T 041ffbedf2c04dbea4661b4c33f78ac1 P b2835e59d40a48aea53c5dedb3bd1e9e reported cstate change: config changed from index 532 to 533, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "2cf1cac003b44643a2d4de1204aacf0f" committed_config { opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.077718 26268 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb [term 1 FOLLOWER]: Committing config change with OpId 1.250: config changed from index 249 to 250, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.080427 25851 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 041ffbedf2c04dbea4661b4c33f78ac1 with cas_config_opid_index 532: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:59.083261 25866 catalog_manager.cc:5484] T 39d3ba2cbbad40d6a232e534c65d2d74 P 7c384005cb3041a0bd70e163473b6acb reported cstate change: config changed from index 249 to 250, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "cff7b7a19fc64db5849391b1d8e10cce" committed_config { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.086902 26117 tablet_service.cc:1514] Processing DeleteTablet for tablet 39d3ba2cbbad40d6a232e534c65d2d74 with delete_type TABLET_DATA_TOMBSTONED (TS 2cf1cac003b44643a2d4de1204aacf0f not found in new config with opid_index 250) from {username='slave'} at 127.0.0.1:54956
I20240503 02:12:59.087169 25988 tablet_service.cc:1514] Processing DeleteTablet for tablet 041ffbedf2c04dbea4661b4c33f78ac1 with delete_type TABLET_DATA_TOMBSTONED (TS 3bb103b2fd2344e7b55caf7389acf221 not found in new config with opid_index 533) from {username='slave'} at 127.0.0.1:43226
I20240503 02:12:59.089684 29878 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Committing config change with OpId 1.250: config changed from index 249 to 250, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.091960 27594 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Committing config change with OpId 1.250: config changed from index 249 to 250, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.098413 26007 raft_consensus.cc:2932] T 39d3ba2cbbad40d6a232e534c65d2d74 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Committing config change with OpId 1.250: config changed from index 249 to 250, VOTER 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194) evicted. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.099324 30755 tablet_replica.cc:330] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221: stopping tablet replica
I20240503 02:12:59.099509 30755 raft_consensus.cc:2229] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.099712 30755 raft_consensus.cc:2258] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.090482 26391 raft_consensus.cc:2932] T 041ffbedf2c04dbea4661b4c33f78ac1 P cff7b7a19fc64db5849391b1d8e10cce [term 1 FOLLOWER]: Committing config change with OpId 1.533: config changed from index 532 to 533, VOTER 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) evicted. New config: { opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.103929 30755 ts_tablet_manager.cc:1875] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:59.110165 30755 ts_tablet_manager.cc:1888] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.532
I20240503 02:12:59.110402 30755 log.cc:1196] T 041ffbedf2c04dbea4661b4c33f78ac1 P 3bb103b2fd2344e7b55caf7389acf221: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/041ffbedf2c04dbea4661b4c33f78ac1
I20240503 02:12:59.111590 25851 catalog_manager.cc:4830] TS 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): tablet 041ffbedf2c04dbea4661b4c33f78ac1 (table rebalance_test_table_1 [id=5c0fbc6200584276a70cc88ce038d3c1]) successfully deleted
I20240503 02:12:59.112124 30754 tablet_replica.cc:330] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f: stopping tablet replica
I20240503 02:12:59.112396 30754 raft_consensus.cc:2229] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.112768 30754 raft_consensus.cc:2258] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.114452 30754 ts_tablet_manager.cc:1875] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:59.120748 30754 ts_tablet_manager.cc:1888] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.249
I20240503 02:12:59.120857 30754 log.cc:1196] T 39d3ba2cbbad40d6a232e534c65d2d74 P 2cf1cac003b44643a2d4de1204aacf0f: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/39d3ba2cbbad40d6a232e534c65d2d74
I20240503 02:12:59.121564 25851 catalog_manager.cc:4830] TS 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): tablet 39d3ba2cbbad40d6a232e534c65d2d74 (table rebalance_test_table_2 [id=d4abb1a36f9a48a3b886746b377881e5]) successfully deleted
I20240503 02:12:59.312101 26656 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 4 FOLLOWER]: Refusing update from remote peer 0a801f0897884f108cdb89434b18628a: Log matching property violated. Preceding OpId in replica: term: 3 index: 497. Preceding OpId from leader: term: 4 index: 498. (index mismatch)
I20240503 02:12:59.312681 30645 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 498, Last known committed idx: 497, Time since last communication: 0.000s
I20240503 02:12:59.363921 29878 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964 [term 4 FOLLOWER]: Refusing update from remote peer 0a801f0897884f108cdb89434b18628a: Log matching property violated. Preceding OpId in replica: term: 3 index: 497. Preceding OpId from leader: term: 4 index: 498. (index mismatch)
I20240503 02:12:59.364413 30645 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 498, Last known committed idx: 497, Time since last communication: 0.000s
I20240503 02:12:59.367934 26268 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 4 FOLLOWER]: Refusing update from remote peer 0a801f0897884f108cdb89434b18628a: Log matching property violated. Preceding OpId in replica: term: 3 index: 497. Preceding OpId from leader: term: 4 index: 498. (index mismatch)
I20240503 02:12:59.368396 30645 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 498, Last known committed idx: 497, Time since last communication: 0.000s
I20240503 02:12:59.372968 26136 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 4 FOLLOWER]: Refusing update from remote peer 0a801f0897884f108cdb89434b18628a: Log matching property violated. Preceding OpId in replica: term: 3 index: 497. Preceding OpId from leader: term: 4 index: 498. (index mismatch)
I20240503 02:12:59.373251 26391 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 4 FOLLOWER]: Refusing update from remote peer 0a801f0897884f108cdb89434b18628a: Log matching property violated. Preceding OpId in replica: term: 3 index: 497. Preceding OpId from leader: term: 4 index: 498. (index mismatch)
I20240503 02:12:59.374208 30645 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 498, Last known committed idx: 497, Time since last communication: 0.000s
I20240503 02:12:59.374666 30645 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 498, Last known committed idx: 497, Time since last communication: 0.000s
I20240503 02:12:59.912940 27594 consensus_queue.cc:237] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 498, Committed index: 498, Last appended: 4.498, Last appended by leader: 497, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:12:59.914868 26268 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 4 FOLLOWER]: Refusing update from remote peer 0a801f0897884f108cdb89434b18628a: Log matching property violated. Preceding OpId in replica: term: 4 index: 498. Preceding OpId from leader: term: 4 index: 499. (index mismatch)
I20240503 02:12:59.915026 26136 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 4 FOLLOWER]: Refusing update from remote peer 0a801f0897884f108cdb89434b18628a: Log matching property violated. Preceding OpId in replica: term: 4 index: 498. Preceding OpId from leader: term: 4 index: 499. (index mismatch)
I20240503 02:12:59.915030 29878 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964 [term 4 FOLLOWER]: Refusing update from remote peer 0a801f0897884f108cdb89434b18628a: Log matching property violated. Preceding OpId in replica: term: 4 index: 498. Preceding OpId from leader: term: 4 index: 499. (index mismatch)
I20240503 02:12:59.915223 26391 raft_consensus.cc:1261] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 4 FOLLOWER]: Refusing update from remote peer 0a801f0897884f108cdb89434b18628a: Log matching property violated. Preceding OpId in replica: term: 4 index: 498. Preceding OpId from leader: term: 4 index: 499. (index mismatch)
I20240503 02:12:59.915720 30744 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 499, Last known committed idx: 498, Time since last communication: 0.000s
I20240503 02:12:59.916270 30799 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 499, Last known committed idx: 498, Time since last communication: 0.000s
I20240503 02:12:59.916697 30783 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 499, Last known committed idx: 498, Time since last communication: 0.000s
I20240503 02:12:59.916987 30645 consensus_queue.cc:1035] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 499, Last known committed idx: 498, Time since last communication: 0.000s
I20240503 02:12:59.919507 30645 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a [term 4 LEADER]: Committing config change with OpId 4.499: config changed from index 497 to 499, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New config: { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.920796 26136 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 2cf1cac003b44643a2d4de1204aacf0f [term 4 FOLLOWER]: Committing config change with OpId 4.499: config changed from index 497 to 499, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New config: { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.922762 25851 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4a864ac954e046eda8925f5daf4a0f1f with cas_config_opid_index 497: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:59.923413 26391 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce [term 4 FOLLOWER]: Committing config change with OpId 4.499: config changed from index 497 to 499, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New config: { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.923472 26268 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 7c384005cb3041a0bd70e163473b6acb [term 4 FOLLOWER]: Committing config change with OpId 4.499: config changed from index 497 to 499, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New config: { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.927362 29878 raft_consensus.cc:2932] T 4a864ac954e046eda8925f5daf4a0f1f P 1b4b70e4a9d54c909b06663e702d4964 [term 4 FOLLOWER]: Committing config change with OpId 4.499: config changed from index 497 to 499, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New config: { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.928347 25862 catalog_manager.cc:5484] T 4a864ac954e046eda8925f5daf4a0f1f P cff7b7a19fc64db5849391b1d8e10cce reported cstate change: config changed from index 497 to 499, VOTER ae8b30bcfa5b425782a95374787dd230 (127.23.104.198) evicted. New cstate: current_term: 4 leader_uuid: "0a801f0897884f108cdb89434b18628a" committed_config { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } } }
I20240503 02:12:59.934850 26638 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a864ac954e046eda8925f5daf4a0f1f with delete_type TABLET_DATA_TOMBSTONED (TS ae8b30bcfa5b425782a95374787dd230 not found in new config with opid_index 499) from {username='slave'} at 127.0.0.1:56314
I20240503 02:12:59.935482 30804 tablet_replica.cc:330] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230: stopping tablet replica
I20240503 02:12:59.935714 30804 raft_consensus.cc:2229] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.936024 30804 raft_consensus.cc:2258] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.938084 30804 ts_tablet_manager.cc:1875] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:59.942677 30804 ts_tablet_manager.cc:1888] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.498
I20240503 02:12:59.942797 30804 log.cc:1196] T 4a864ac954e046eda8925f5daf4a0f1f P ae8b30bcfa5b425782a95374787dd230: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal/wals/4a864ac954e046eda8925f5daf4a0f1f
I20240503 02:12:59.943475 25854 catalog_manager.cc:4830] TS ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369): tablet 4a864ac954e046eda8925f5daf4a0f1f (table rebalance_test_table_0 [id=139fcfbed25b4601a8192d5ddba81e27]) successfully deleted
I20240503 02:13:00.167719 27573 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:00.172917 25988 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:00.176677 29860 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:00.189707 26117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:13:00.221402 26638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:00.237720 26247 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:00.254868 26509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:00.286059 26378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:00.506175 27573 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:00.532552 26117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:13:00.539304 25988 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:00.567379 29860 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:00.569689 26509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:00.570262 26638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:00.588866 26378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:00.601261 26247 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:00.950588 27573 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:00.959681 29860 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:00.963578 25988 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:00.972664 26247 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:00.973932 26117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:13:00.989871 26638 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:01.021415 26509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:01.052882 26378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b8f9a407d2264aa2ba34e0ddd69ece11 | 127.23.104.254:40709 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.23.104.212:46007 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
0a801f0897884f108cdb89434b18628a | 127.23.104.199:34279 | HEALTHY | <none> | 1 | 0
1b4b70e4a9d54c909b06663e702d4964 | 127.23.104.200:34857 | HEALTHY | <none> | 0 | 0
2cf1cac003b44643a2d4de1204aacf0f | 127.23.104.194:39753 | HEALTHY | <none> | 4 | 0
3bb103b2fd2344e7b55caf7389acf221 | 127.23.104.193:34143 | HEALTHY | <none> | 2 | 0
7c384005cb3041a0bd70e163473b6acb | 127.23.104.195:45999 | HEALTHY | <none> | 3 | 0
ae8b30bcfa5b425782a95374787dd230 | 127.23.104.198:40369 | HEALTHY | <none> | 1 | 0
b2835e59d40a48aea53c5dedb3bd1e9e | 127.23.104.197:38347 | HEALTHY | <none> | 2 | 0
cff7b7a19fc64db5849391b1d8e10cce | 127.23.104.196:43675 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.23.104.193 | experimental | 127.23.104.193:34143
local_ip_for_outbound_sockets | 127.23.104.194 | experimental | 127.23.104.194:39753
local_ip_for_outbound_sockets | 127.23.104.195 | experimental | 127.23.104.195:45999
local_ip_for_outbound_sockets | 127.23.104.196 | experimental | 127.23.104.196:43675
local_ip_for_outbound_sockets | 127.23.104.197 | experimental | 127.23.104.197:38347
local_ip_for_outbound_sockets | 127.23.104.198 | experimental | 127.23.104.198:40369
local_ip_for_outbound_sockets | 127.23.104.199 | experimental | 127.23.104.199:34279
local_ip_for_outbound_sockets | 127.23.104.200 | experimental | 127.23.104.200:34857
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.104.193:34143
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.23.104.194:39753
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.104.195:45999
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.23.104.196:43675
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.23.104.197:38347
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.23.104.198:40369
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.23.104.199:34279
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.23.104.200:34857
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.23.104.212:46007 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 02:13:01.115304 23971 log_verifier.cc:126] Checking tablet 041ffbedf2c04dbea4661b4c33f78ac1
I20240503 02:13:01.610371 23971 log_verifier.cc:177] Verified matching terms for 533 ops in tablet 041ffbedf2c04dbea4661b4c33f78ac1
I20240503 02:13:01.611637 23971 log_verifier.cc:126] Checking tablet 1162f9075da248d0a919db2eb45aa949
I20240503 02:13:01.998920 23971 log_verifier.cc:177] Verified matching terms for 501 ops in tablet 1162f9075da248d0a919db2eb45aa949
I20240503 02:13:02.000109 23971 log_verifier.cc:126] Checking tablet 21e7ddb56be1499b94920ebd252595ce
I20240503 02:13:02.450431 23971 log_verifier.cc:177] Verified matching terms for 517 ops in tablet 21e7ddb56be1499b94920ebd252595ce
I20240503 02:13:02.451746 23971 log_verifier.cc:126] Checking tablet 39d3ba2cbbad40d6a232e534c65d2d74
I20240503 02:13:02.698482 23971 log_verifier.cc:177] Verified matching terms for 250 ops in tablet 39d3ba2cbbad40d6a232e534c65d2d74
I20240503 02:13:02.699208 23971 log_verifier.cc:126] Checking tablet 495c99a8850d4ec6b52a89bec172bab0
I20240503 02:13:03.116454 23971 log_verifier.cc:177] Verified matching terms for 518 ops in tablet 495c99a8850d4ec6b52a89bec172bab0
I20240503 02:13:03.117923 23971 log_verifier.cc:126] Checking tablet 4a864ac954e046eda8925f5daf4a0f1f
I20240503 02:13:03.533705 23971 log_verifier.cc:177] Verified matching terms for 499 ops in tablet 4a864ac954e046eda8925f5daf4a0f1f
I20240503 02:13:03.534890 23971 log_verifier.cc:126] Checking tablet 508ee4b913dd40b49d50be3fa4cdd6c4
I20240503 02:13:03.975419 23971 log_verifier.cc:177] Verified matching terms for 522 ops in tablet 508ee4b913dd40b49d50be3fa4cdd6c4
I20240503 02:13:03.976526 23971 log_verifier.cc:126] Checking tablet 5332ecb1e4454eadb47cc721e2fbc487
I20240503 02:13:04.005776 23971 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 5332ecb1e4454eadb47cc721e2fbc487
I20240503 02:13:04.005869 23971 log_verifier.cc:126] Checking tablet 740218c49aa84ee28f875f2cfcee01b8
I20240503 02:13:04.253666 23971 log_verifier.cc:177] Verified matching terms for 269 ops in tablet 740218c49aa84ee28f875f2cfcee01b8
I20240503 02:13:04.254364 23971 log_verifier.cc:126] Checking tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d
I20240503 02:13:04.773406 23971 log_verifier.cc:177] Verified matching terms for 569 ops in tablet 7eb1a7e7a1a440c1a826ac2f2e27db2d
I20240503 02:13:04.774698 23971 log_verifier.cc:126] Checking tablet c12cc2e0fb09431e95baae55803f2184
I20240503 02:13:05.053622 23971 log_verifier.cc:177] Verified matching terms for 269 ops in tablet c12cc2e0fb09431e95baae55803f2184
I20240503 02:13:05.054284 23971 log_verifier.cc:126] Checking tablet c27502976d474b938cce2235e5784f79
I20240503 02:13:05.307166 23971 log_verifier.cc:177] Verified matching terms for 283 ops in tablet c27502976d474b938cce2235e5784f79
I20240503 02:13:05.308147 23971 log_verifier.cc:126] Checking tablet c3b6c051161d4f9ba9d1d38a06a4c633
I20240503 02:13:05.698734 23971 log_verifier.cc:177] Verified matching terms for 488 ops in tablet c3b6c051161d4f9ba9d1d38a06a4c633
I20240503 02:13:05.700117 23971 log_verifier.cc:126] Checking tablet d972dd11fcd9484f9a59f63d89f1e6e2
I20240503 02:13:06.194145 23971 log_verifier.cc:177] Verified matching terms for 525 ops in tablet d972dd11fcd9484f9a59f63d89f1e6e2
I20240503 02:13:06.195379 23971 log_verifier.cc:126] Checking tablet e866b4f1b338419ab1025287b2aaac5e
I20240503 02:13:06.456925 23971 log_verifier.cc:177] Verified matching terms for 277 ops in tablet e866b4f1b338419ab1025287b2aaac5e
I20240503 02:13:06.457574 23971 log_verifier.cc:126] Checking tablet fa2fbede85374d1492edeaafef243747
I20240503 02:13:06.687610 23971 log_verifier.cc:177] Verified matching terms for 269 ops in tablet fa2fbede85374d1492edeaafef243747
I20240503 02:13:06.713542 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 25926
I20240503 02:13:06.724328 26048 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:13:07.009652 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 25926
W20240503 02:13:07.056275 26462 connection.cc:535] server connection from 127.23.104.193:55847 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:07.056423 26202 connection.cc:535] client connection to 127.23.104.193:34143 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:13:07.056574 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 26057
W20240503 02:13:07.056660 26202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:07.057895 26332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.104.193:34143 (error 108)
W20240503 02:13:07.060024 26332 consensus_peers.cc:485] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce -> Peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Couldn't send request to peer 3bb103b2fd2344e7b55caf7389acf221. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:07.060134 26202 consensus_peers.cc:485] T c27502976d474b938cce2235e5784f79 P 7c384005cb3041a0bd70e163473b6acb -> Peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Couldn't send request to peer 3bb103b2fd2344e7b55caf7389acf221. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:07.062686 26178 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:13:07.181187 26202 consensus_peers.cc:485] T 21e7ddb56be1499b94920ebd252595ce P 7c384005cb3041a0bd70e163473b6acb -> Peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Couldn't send request to peer 3bb103b2fd2344e7b55caf7389acf221. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:07.297993 26332 consensus_peers.cc:485] T c3b6c051161d4f9ba9d1d38a06a4c633 P cff7b7a19fc64db5849391b1d8e10cce -> Peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Couldn't send request to peer 3bb103b2fd2344e7b55caf7389acf221. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:07.346709 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 26057
W20240503 02:13:07.391772 26332 consensus_peers.cc:485] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce -> Peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Couldn't send request to peer 3bb103b2fd2344e7b55caf7389acf221. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:07.392136 27527 connection.cc:535] server connection from 127.23.104.194:42389 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:07.392249 27524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.104.194:39753 (error 108)
W20240503 02:13:07.392431 26595 connection.cc:535] server connection from 127.23.104.194:55525 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:07.392619 26202 connection.cc:535] client connection to 127.23.104.194:39753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:07.392951 26464 connection.cc:535] server connection from 127.23.104.194:51223 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:13:07.393455 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 26187
W20240503 02:13:07.393566 26202 consensus_peers.cc:485] T 740218c49aa84ee28f875f2cfcee01b8 P 7c384005cb3041a0bd70e163473b6acb -> Peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): Couldn't send request to peer 2cf1cac003b44643a2d4de1204aacf0f. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.194:39753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:07.394595 27524 consensus_peers.cc:485] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a -> Peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): Couldn't send request to peer 2cf1cac003b44643a2d4de1204aacf0f. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.194:39753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:07.403379 26592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111)
I20240503 02:13:07.403679 26308 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:13:07.404685 26592 consensus_peers.cc:485] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 -> Peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Couldn't send request to peer 3bb103b2fd2344e7b55caf7389acf221. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:07.517102 26332 consensus_peers.cc:485] T d972dd11fcd9484f9a59f63d89f1e6e2 P cff7b7a19fc64db5849391b1d8e10cce -> Peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): Couldn't send request to peer 2cf1cac003b44643a2d4de1204aacf0f. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.194:39753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:07.670187 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 26187
W20240503 02:13:07.717099 26595 connection.cc:535] server connection from 127.23.104.195:34683 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:07.717589 26334 connection.cc:535] client connection to 127.23.104.195:45999 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:07.717629 26462 connection.cc:535] server connection from 127.23.104.195:46953 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:07.717629 26464 connection.cc:535] client connection to 127.23.104.195:45999 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:13:07.717787 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 26317
W20240503 02:13:07.717952 26464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:07.719240 26334 consensus_peers.cc:485] T 39d3ba2cbbad40d6a232e534c65d2d74 P cff7b7a19fc64db5849391b1d8e10cce -> Peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999): Couldn't send request to peer 7c384005cb3041a0bd70e163473b6acb. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.195:45999: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:07.719702 26464 consensus_peers.cc:485] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e -> Peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999): Couldn't send request to peer 7c384005cb3041a0bd70e163473b6acb. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.195:45999: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:07.719830 26464 consensus_peers.cc:485] T 508ee4b913dd40b49d50be3fa4cdd6c4 P b2835e59d40a48aea53c5dedb3bd1e9e -> Peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999): Couldn't send request to peer 7c384005cb3041a0bd70e163473b6acb. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.195:45999: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:07.720541 26462 consensus_peers.cc:485] T 1162f9075da248d0a919db2eb45aa949 P b2835e59d40a48aea53c5dedb3bd1e9e -> Peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): Couldn't send request to peer 2cf1cac003b44643a2d4de1204aacf0f. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.194:39753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:07.725785 26438 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:13:08.006129 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 26317
W20240503 02:13:08.065115 26465 connection.cc:535] server connection from 127.23.104.196:48555 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:13:08.066030 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 26447
I20240503 02:13:08.071100 26569 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:13:08.079249 27526 consensus_peers.cc:485] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a -> Peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999): Couldn't send request to peer 7c384005cb3041a0bd70e163473b6acb. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.195:45999: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:08.107151 30986 raft_consensus.cc:482] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2cf1cac003b44643a2d4de1204aacf0f)
I20240503 02:13:08.107404 30986 raft_consensus.cc:504] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:13:08.109287 30986 leader_election.cc:288] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347)
I20240503 02:13:08.110312 26654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041ffbedf2c04dbea4661b4c33f78ac1" candidate_uuid: "1b4b70e4a9d54c909b06663e702d4964" candidate_term: 2 candidate_status { last_received { term: 1 index: 533 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
W20240503 02:13:08.110961 29815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.104.196:43675: connect: Connection refused (error 111)
W20240503 02:13:08.112701 29813 leader_election.cc:334] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): Network error: Client connection negotiation failed: client connection to 127.23.104.194:39753: connect: Connection refused (error 111)
W20240503 02:13:08.112879 29815 leader_election.cc:334] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675): Network error: Client connection negotiation failed: client connection to 127.23.104.196:43675: connect: Connection refused (error 111)
I20240503 02:13:08.113013 29815 leader_election.cc:302] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 1b4b70e4a9d54c909b06663e702d4964; no voters: 2cf1cac003b44643a2d4de1204aacf0f, ae8b30bcfa5b425782a95374787dd230, cff7b7a19fc64db5849391b1d8e10cce
I20240503 02:13:08.113344 30986 raft_consensus.cc:2726] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:08.160835 30973 raft_consensus.cc:482] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3bb103b2fd2344e7b55caf7389acf221)
I20240503 02:13:08.161013 30973 raft_consensus.cc:504] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:13:08.162021 30973 leader_election.cc:288] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857)
I20240503 02:13:08.162458 26654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "0a801f0897884f108cdb89434b18628a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:13:08.162676 29878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "0a801f0897884f108cdb89434b18628a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b4b70e4a9d54c909b06663e702d4964" is_pre_election: true
W20240503 02:13:08.163403 27524 leader_election.cc:334] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111)
I20240503 02:13:08.163512 27524 leader_election.cc:302] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 0a801f0897884f108cdb89434b18628a; no voters: 1b4b70e4a9d54c909b06663e702d4964, 3bb103b2fd2344e7b55caf7389acf221, ae8b30bcfa5b425782a95374787dd230
I20240503 02:13:08.163750 30973 raft_consensus.cc:2726] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:08.187849 30979 raft_consensus.cc:482] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3bb103b2fd2344e7b55caf7389acf221)
I20240503 02:13:08.188055 30979 raft_consensus.cc:504] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:13:08.188966 30979 leader_election.cc:288] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857)
I20240503 02:13:08.189498 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a801f0897884f108cdb89434b18628a" is_pre_election: true
I20240503 02:13:08.189594 29878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b4b70e4a9d54c909b06663e702d4964" is_pre_election: true
I20240503 02:13:08.189746 27591 raft_consensus.cc:2454] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 1.
W20240503 02:13:08.190366 26592 leader_election.cc:334] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111)
I20240503 02:13:08.197608 30986 raft_consensus.cc:482] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3bb103b2fd2344e7b55caf7389acf221)
I20240503 02:13:08.197768 30986 raft_consensus.cc:504] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:13:08.199232 26654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "1b4b70e4a9d54c909b06663e702d4964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:13:08.199331 30986 leader_election.cc:288] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369)
I20240503 02:13:08.199430 26654 raft_consensus.cc:2454] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b4b70e4a9d54c909b06663e702d4964 in term 1.
W20240503 02:13:08.201020 29813 leader_election.cc:334] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111)
I20240503 02:13:08.204524 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "1b4b70e4a9d54c909b06663e702d4964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a801f0897884f108cdb89434b18628a" is_pre_election: true
I20240503 02:13:08.204696 27591 raft_consensus.cc:2454] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b4b70e4a9d54c909b06663e702d4964 in term 1.
I20240503 02:13:08.205045 29813 leader_election.cc:302] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 0a801f0897884f108cdb89434b18628a, 1b4b70e4a9d54c909b06663e702d4964, ae8b30bcfa5b425782a95374787dd230; no voters: 3bb103b2fd2344e7b55caf7389acf221
I20240503 02:13:08.205307 30986 raft_consensus.cc:2781] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:08.205426 30986 raft_consensus.cc:482] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3bb103b2fd2344e7b55caf7389acf221)
I20240503 02:13:08.205529 30986 raft_consensus.cc:3037] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:08.208590 30986 raft_consensus.cc:504] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:13:08.209606 30986 leader_election.cc:288] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 election: Requested vote from peers 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369)
I20240503 02:13:08.209924 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "1b4b70e4a9d54c909b06663e702d4964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a801f0897884f108cdb89434b18628a"
I20240503 02:13:08.210098 27591 raft_consensus.cc:3037] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:08.210072 26654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "1b4b70e4a9d54c909b06663e702d4964" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230"
I20240503 02:13:08.210283 26654 raft_consensus.cc:3037] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:08.212859 27591 raft_consensus.cc:2454] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4b70e4a9d54c909b06663e702d4964 in term 2.
I20240503 02:13:08.213135 26654 raft_consensus.cc:2454] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4b70e4a9d54c909b06663e702d4964 in term 2.
I20240503 02:13:08.213837 29813 leader_election.cc:302] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0a801f0897884f108cdb89434b18628a, 1b4b70e4a9d54c909b06663e702d4964, ae8b30bcfa5b425782a95374787dd230; no voters:
I20240503 02:13:08.214121 30986 raft_consensus.cc:2781] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [term 2 FOLLOWER]: Leader election won for term 2
W20240503 02:13:08.215495 29813 leader_election.cc:334] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111)
I20240503 02:13:08.216002 30986 raft_consensus.cc:686] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [term 2 LEADER]: Becoming Leader. State: Replica: 1b4b70e4a9d54c909b06663e702d4964, State: Running, Role: LEADER
I20240503 02:13:08.216611 30986 consensus_queue.cc:237] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:13:08.225412 25858 catalog_manager.cc:5484] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 reported cstate change: term changed from 1 to 2, leader changed from 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193) to 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200). New cstate: current_term: 2 leader_uuid: "1b4b70e4a9d54c909b06663e702d4964" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 02:13:08.242671 27526 consensus_peers.cc:485] T 4a864ac954e046eda8925f5daf4a0f1f P 0a801f0897884f108cdb89434b18628a -> Peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675): Couldn't send request to peer cff7b7a19fc64db5849391b1d8e10cce. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.196:43675: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:08.266685 30973 raft_consensus.cc:482] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2cf1cac003b44643a2d4de1204aacf0f)
I20240503 02:13:08.266891 30973 raft_consensus.cc:504] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } }
I20240503 02:13:08.267745 30973 leader_election.cc:288] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
W20240503 02:13:08.269281 27526 leader_election.cc:334] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999): Network error: Client connection negotiation failed: client connection to 127.23.104.195:45999: connect: Connection refused (error 111)
W20240503 02:13:08.269531 27526 leader_election.cc:334] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675): Network error: Client connection negotiation failed: client connection to 127.23.104.196:43675: connect: Connection refused (error 111)
W20240503 02:13:08.270218 27524 leader_election.cc:334] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111)
I20240503 02:13:08.270306 27524 leader_election.cc:302] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 0a801f0897884f108cdb89434b18628a; no voters: 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb, cff7b7a19fc64db5849391b1d8e10cce
I20240503 02:13:08.270517 30973 raft_consensus.cc:2726] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:13:08.270530 27524 leader_election.cc:334] T 495c99a8850d4ec6b52a89bec172bab0 P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): Network error: Client connection negotiation failed: client connection to 127.23.104.194:39753: connect: Connection refused (error 111)
I20240503 02:13:08.350011 26528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "0a801f0897884f108cdb89434b18628a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:13:08.350512 26523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332ecb1e4454eadb47cc721e2fbc487" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" is_pre_election: true
I20240503 02:13:08.351346 26595 leader_election.cc:302] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 0a801f0897884f108cdb89434b18628a, ae8b30bcfa5b425782a95374787dd230; no voters: 1b4b70e4a9d54c909b06663e702d4964, 3bb103b2fd2344e7b55caf7389acf221, b2835e59d40a48aea53c5dedb3bd1e9e
I20240503 02:13:08.351686 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 26447
I20240503 02:13:08.352092 30979 raft_consensus.cc:2726] T 5332ecb1e4454eadb47cc721e2fbc487 P ae8b30bcfa5b425782a95374787dd230 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:08.383204 30978 raft_consensus.cc:482] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2cf1cac003b44643a2d4de1204aacf0f)
I20240503 02:13:08.383390 30978 raft_consensus.cc:504] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:13:08.384294 30978 leader_election.cc:288] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857)
I20240503 02:13:08.385159 29878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041ffbedf2c04dbea4661b4c33f78ac1" candidate_uuid: "ae8b30bcfa5b425782a95374787dd230" candidate_term: 2 candidate_status { last_received { term: 1 index: 533 } } ignore_live_leader: false dest_uuid: "1b4b70e4a9d54c909b06663e702d4964" is_pre_election: true
I20240503 02:13:08.385430 29878 raft_consensus.cc:2454] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8b30bcfa5b425782a95374787dd230 in term 1.
W20240503 02:13:08.386049 26592 leader_election.cc:334] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): Network error: Client connection negotiation failed: client connection to 127.23.104.194:39753: connect: Connection refused (error 111)
W20240503 02:13:08.386261 26594 leader_election.cc:334] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675): Network error: Client connection negotiation failed: client connection to 127.23.104.196:43675: connect: Connection refused (error 111)
I20240503 02:13:08.395650 30986 raft_consensus.cc:482] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7c384005cb3041a0bd70e163473b6acb)
I20240503 02:13:08.395879 30986 raft_consensus.cc:504] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:13:08.396842 30986 leader_election.cc:288] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279)
I20240503 02:13:08.397118 26654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27502976d474b938cce2235e5784f79" candidate_uuid: "1b4b70e4a9d54c909b06663e702d4964" candidate_term: 2 candidate_status { last_received { term: 1 index: 283 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:13:08.398365 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27502976d474b938cce2235e5784f79" candidate_uuid: "1b4b70e4a9d54c909b06663e702d4964" candidate_term: 2 candidate_status { last_received { term: 1 index: 283 } } ignore_live_leader: false dest_uuid: "0a801f0897884f108cdb89434b18628a" is_pre_election: true
W20240503 02:13:08.398597 29815 leader_election.cc:334] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999): Network error: Client connection negotiation failed: client connection to 127.23.104.195:45999: connect: Connection refused (error 111)
W20240503 02:13:08.398788 29813 leader_election.cc:334] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111)
I20240503 02:13:08.399068 29813 leader_election.cc:302] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 1b4b70e4a9d54c909b06663e702d4964; no voters: 0a801f0897884f108cdb89434b18628a, 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb
I20240503 02:13:08.399565 30986 raft_consensus.cc:2726] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:13:08.402020 29816 connection.cc:535] client connection to 127.23.104.197:38347 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:08.402155 26595 connection.cc:535] client connection to 127.23.104.197:38347 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:13:08.402839 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 26577
W20240503 02:13:08.403174 26595 leader_election.cc:334] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347): Network error: Client connection negotiation failed: client connection to 127.23.104.197:38347: connect: Connection refused (error 111)
I20240503 02:13:08.403297 26595 leader_election.cc:302] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 1b4b70e4a9d54c909b06663e702d4964, ae8b30bcfa5b425782a95374787dd230; no voters: 2cf1cac003b44643a2d4de1204aacf0f, b2835e59d40a48aea53c5dedb3bd1e9e, cff7b7a19fc64db5849391b1d8e10cce
W20240503 02:13:08.403386 29816 leader_election.cc:334] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347): Network error: Client connection negotiation failed: client connection to 127.23.104.197:38347: connect: Connection refused (error 111)
W20240503 02:13:08.403692 29816 leader_election.cc:334] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347): Network error: Client connection negotiation failed: client connection to 127.23.104.197:38347: connect: Connection refused (error 111)
W20240503 02:13:08.403666 27527 leader_election.cc:334] T 5332ecb1e4454eadb47cc721e2fbc487 P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347): Network error: Client connection negotiation failed: client connection to 127.23.104.197:38347: connect: Connection refused (error 111)
I20240503 02:13:08.403872 30978 raft_consensus.cc:2726] T 041ffbedf2c04dbea4661b4c33f78ac1 P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:13:08.403944 29816 leader_election.cc:334] T 041ffbedf2c04dbea4661b4c33f78ac1 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347): Network error: Client connection negotiation failed: client connection to 127.23.104.197:38347: connect: Connection refused (error 111)
I20240503 02:13:08.408427 26699 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:13:08.440402 30973 raft_consensus.cc:482] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7c384005cb3041a0bd70e163473b6acb)
I20240503 02:13:08.440649 30973 raft_consensus.cc:504] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:13:08.441603 30973 leader_election.cc:288] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857)
I20240503 02:13:08.442562 29878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27502976d474b938cce2235e5784f79" candidate_uuid: "0a801f0897884f108cdb89434b18628a" candidate_term: 2 candidate_status { last_received { term: 1 index: 283 } } ignore_live_leader: false dest_uuid: "1b4b70e4a9d54c909b06663e702d4964" is_pre_election: true
I20240503 02:13:08.442821 29878 raft_consensus.cc:2454] T c27502976d474b938cce2235e5784f79 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a801f0897884f108cdb89434b18628a in term 1.
W20240503 02:13:08.443318 27524 leader_election.cc:334] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111)
W20240503 02:13:08.443581 27526 leader_election.cc:334] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999): Network error: Client connection negotiation failed: client connection to 127.23.104.195:45999: connect: Connection refused (error 111)
I20240503 02:13:08.559082 30986 raft_consensus.cc:482] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2cf1cac003b44643a2d4de1204aacf0f)
I20240503 02:13:08.559336 30986 raft_consensus.cc:504] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:13:08.560292 30986 leader_election.cc:288] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279)
W20240503 02:13:08.561992 29815 leader_election.cc:334] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675): Network error: Client connection negotiation failed: client connection to 127.23.104.196:43675: connect: Connection refused (error 111)
I20240503 02:13:08.561980 27591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12cc2e0fb09431e95baae55803f2184" candidate_uuid: "1b4b70e4a9d54c909b06663e702d4964" candidate_term: 2 candidate_status { last_received { term: 1 index: 269 } } ignore_live_leader: false dest_uuid: "0a801f0897884f108cdb89434b18628a" is_pre_election: true
W20240503 02:13:08.562652 29813 leader_election.cc:334] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753): Network error: Client connection negotiation failed: client connection to 127.23.104.194:39753: connect: Connection refused (error 111)
I20240503 02:13:08.562789 29813 leader_election.cc:302] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 1b4b70e4a9d54c909b06663e702d4964; no voters: 0a801f0897884f108cdb89434b18628a, 2cf1cac003b44643a2d4de1204aacf0f, cff7b7a19fc64db5849391b1d8e10cce
W20240503 02:13:08.562976 29816 leader_election.cc:334] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347): Network error: Client connection negotiation failed: client connection to 127.23.104.197:38347: connect: Connection refused (error 111)
I20240503 02:13:08.563088 30986 raft_consensus.cc:2726] T c12cc2e0fb09431e95baae55803f2184 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:13:08.671613 29813 consensus_peers.cc:485] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 -> Peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Couldn't send request to peer 3bb103b2fd2344e7b55caf7389acf221. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:08.679556 26654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c27502976d474b938cce2235e5784f79" candidate_uuid: "0a801f0897884f108cdb89434b18628a" candidate_term: 2 candidate_status { last_received { term: 1 index: 283 } } ignore_live_leader: false dest_uuid: "ae8b30bcfa5b425782a95374787dd230" is_pre_election: true
I20240503 02:13:08.680150 27527 leader_election.cc:302] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 0a801f0897884f108cdb89434b18628a, 1b4b70e4a9d54c909b06663e702d4964; no voters: 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb, ae8b30bcfa5b425782a95374787dd230
I20240503 02:13:08.680305 30979 raft_consensus.cc:482] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3bb103b2fd2344e7b55caf7389acf221)
I20240503 02:13:08.680506 30979 raft_consensus.cc:504] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } attrs { promote: false } }
I20240503 02:13:08.680739 31005 raft_consensus.cc:2726] T c27502976d474b938cce2235e5784f79 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:13:08.681246 26595 consensus_peers.cc:485] T 7eb1a7e7a1a440c1a826ac2f2e27db2d P ae8b30bcfa5b425782a95374787dd230 -> Peer b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347): Couldn't send request to peer b2835e59d40a48aea53c5dedb3bd1e9e. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.197:38347: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:08.681454 30979 leader_election.cc:288] T e866b4f1b338419ab1025287b2aaac5e P ae8b30bcfa5b425782a95374787dd230 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753)
I20240503 02:13:08.681473 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 26577
W20240503 02:13:08.713583 29816 consensus_peers.cc:485] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 -> Peer b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347): Couldn't send request to peer b2835e59d40a48aea53c5dedb3bd1e9e. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.197:38347: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:08.729700 29816 connection.cc:535] client connection to 127.23.104.198:40369 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:13:08.730281 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 27509
W20240503 02:13:08.730937 29816 consensus_peers.cc:485] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 -> Peer ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369): Couldn't send request to peer ae8b30bcfa5b425782a95374787dd230. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.198:40369: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:08.735708 27634 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:13:08.850661 30986 raft_consensus.cc:482] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7c384005cb3041a0bd70e163473b6acb)
I20240503 02:13:08.850921 30986 raft_consensus.cc:504] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "ae8b30bcfa5b425782a95374787dd230" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 40369 } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:13:08.851999 30986 leader_election.cc:288] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369)
W20240503 02:13:08.853482 29813 leader_election.cc:334] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111)
W20240503 02:13:08.853660 29815 leader_election.cc:334] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999): Network error: Client connection negotiation failed: client connection to 127.23.104.195:45999: connect: Connection refused (error 111)
W20240503 02:13:08.854275 29816 leader_election.cc:334] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347): Network error: Client connection negotiation failed: client connection to 127.23.104.197:38347: connect: Connection refused (error 111)
I20240503 02:13:08.854379 29816 leader_election.cc:302] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 1b4b70e4a9d54c909b06663e702d4964; no voters: 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb, b2835e59d40a48aea53c5dedb3bd1e9e
I20240503 02:13:08.854677 30986 raft_consensus.cc:2726] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:13:08.854719 29816 leader_election.cc:334] T 21e7ddb56be1499b94920ebd252595ce P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae8b30bcfa5b425782a95374787dd230 (127.23.104.198:40369): Network error: Client connection negotiation failed: client connection to 127.23.104.198:40369: connect: Connection refused (error 111)
I20240503 02:13:08.924342 30986 raft_consensus.cc:482] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cff7b7a19fc64db5849391b1d8e10cce)
I20240503 02:13:08.924563 30986 raft_consensus.cc:504] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:13:08.925482 30986 leader_election.cc:288] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999), 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143), cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279)
W20240503 02:13:08.926923 29815 leader_election.cc:334] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7c384005cb3041a0bd70e163473b6acb (127.23.104.195:45999): Network error: Client connection negotiation failed: client connection to 127.23.104.195:45999: connect: Connection refused (error 111)
W20240503 02:13:08.927143 29815 leader_election.cc:334] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675): Network error: Client connection negotiation failed: client connection to 127.23.104.196:43675: connect: Connection refused (error 111)
W20240503 02:13:08.927387 29813 leader_election.cc:334] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3bb103b2fd2344e7b55caf7389acf221 (127.23.104.193:34143): Network error: Client connection negotiation failed: client connection to 127.23.104.193:34143: connect: Connection refused (error 111)
I20240503 02:13:08.927564 29813 leader_election.cc:302] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 1b4b70e4a9d54c909b06663e702d4964; no voters: 3bb103b2fd2344e7b55caf7389acf221, 7c384005cb3041a0bd70e163473b6acb, cff7b7a19fc64db5849391b1d8e10cce
I20240503 02:13:08.927856 30986 raft_consensus.cc:2726] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:08.973974 30973 raft_consensus.cc:482] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2cf1cac003b44643a2d4de1204aacf0f)
I20240503 02:13:08.974296 31005 raft_consensus.cc:482] T 740218c49aa84ee28f875f2cfcee01b8 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7c384005cb3041a0bd70e163473b6acb)
I20240503 02:13:08.974612 30973 raft_consensus.cc:504] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "b2835e59d40a48aea53c5dedb3bd1e9e" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 38347 } } peers { permanent_uuid: "2cf1cac003b44643a2d4de1204aacf0f" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 39753 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:13:08.974915 31010 raft_consensus.cc:482] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cff7b7a19fc64db5849391b1d8e10cce)
I20240503 02:13:08.975075 31010 raft_consensus.cc:504] T 39d3ba2cbbad40d6a232e534c65d2d74 P 0a801f0897884f108cdb89434b18628a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "7c384005cb3041a0bd70e163473b6acb" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45999 } } peers { permanent_uuid: "3bb103b2fd2344e7b55caf7389acf221" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 34143 } } peers { permanent_uuid: "cff7b7a19fc64db5849391b1d8e10cce" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 43675 } } peers { permanent_uuid: "0a801f0897884f108cdb89434b18628a" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 34279 } attrs { promote: false } } peers { permanent_uuid: "1b4b70e4a9d54c909b06663e702d4964" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 34857 } attrs { promote: false } }
I20240503 02:13:08.975759 30973 leader_election.cc:288] T c12cc2e0fb09431e95baae55803f2184 P 0a801f0897884f108cdb89434b18628a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cff7b7a19fc64db5849391b1d8e10cce (127.23.104.196:43675), b2835e59d40a48aea53c5dedb3bd1e9e (127.23.104.197:38347), 2cf1cac003b44643a2d4de1204aacf0f (127.23.104.194:39753), 1b4b70e4a9d54c909b06663e702d4964 (127.23.104.200:34857)
I20240503 02:13:08.975970 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 27509
W20240503 02:13:09.013839 29813 connection.cc:535] client connection to 127.23.104.199:34279 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:13:09.014173 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 29797
W20240503 02:13:09.015197 29813 consensus_peers.cc:485] T 5332ecb1e4454eadb47cc721e2fbc487 P 1b4b70e4a9d54c909b06663e702d4964 -> Peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Couldn't send request to peer 0a801f0897884f108cdb89434b18628a. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.199:34279: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:09.015352 29813 leader_election.cc:334] T 39d3ba2cbbad40d6a232e534c65d2d74 P 1b4b70e4a9d54c909b06663e702d4964 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a801f0897884f108cdb89434b18628a (127.23.104.199:34279): Network error: Client connection negotiation failed: client connection to 127.23.104.199:34279: connect: Connection refused (error 111)
I20240503 02:13:09.019884 29920 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:13:09.187155 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 29797
I20240503 02:13:09.215337 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 25836
I20240503 02:13:09.216537 25896 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:13:09.392715 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 25836
2024-05-03T02:13:09Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/7 (89866 ms)
[----------] 1 test from RebalanceParamTest (89866 ms total)
[----------] 1 test from TserverAddedDuringRebalancingTest
[ RUN ] TserverAddedDuringRebalancingTest.TserverStarts/1
I20240503 02:13:09.540769 23971 test_util.cc:268] Using random seed: 1121122182
I20240503 02:13:09.542250 23971 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:13:09.542312 23971 ts_itest-base.cc:113] --------------
I20240503 02:13:09.542358 23971 ts_itest-base.cc:114] 8 tablet servers
I20240503 02:13:09.542405 23971 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:13:09.542455 23971 ts_itest-base.cc:116] --------------
2024-05-03T02:13:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:13:09Z Disabled control of system clock
I20240503 02:13:09.589617 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.104.254:43173
--webserver_interface=127.23.104.254
--webserver_port=0
--builtin_ntp_servers=127.23.104.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.104.254:43173
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 02:13:09.906489 31021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:09.906781 31021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:09.906908 31021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:09.914008 31021 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:09.914119 31021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:13:09.914193 31021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:09.914270 31021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:13:09.914373 31021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:13:09.922883 31021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.23.104.254:43173
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.104.254:43173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.23.104.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:09.924031 31021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:09.925729 31021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:09.935840 31026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:09.936570 31029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:09.937215 31027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:09.937922 31021 server_base.cc:1034] running on GCE node
I20240503 02:13:09.938614 31021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:09.939920 31021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:09.941114 31021 hybrid_clock.cc:648] HybridClock initialized: now 1714702389941071 us; error 35 us; skew 500 ppm
I20240503 02:13:09.941421 31021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:09.943580 31021 webserver.cc:460] Webserver started at http://127.23.104.254:45553/ using document root <none> and password file <none>
I20240503 02:13:09.944252 31021 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:09.944404 31021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:09.944715 31021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:09.947103 31021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "32ac8e5f9d824ced889084d36419cdf5"
format_stamp: "Formatted at 2024-05-03 02:13:09 on dist-test-slave-38ht"
I20240503 02:13:09.947842 31021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "32ac8e5f9d824ced889084d36419cdf5"
format_stamp: "Formatted at 2024-05-03 02:13:09 on dist-test-slave-38ht"
I20240503 02:13:09.952508 31021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240503 02:13:09.955482 31035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:09.956679 31021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:13:09.956836 31021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
uuid: "32ac8e5f9d824ced889084d36419cdf5"
format_stamp: "Formatted at 2024-05-03 02:13:09 on dist-test-slave-38ht"
I20240503 02:13:09.957015 31021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:09.977756 31021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:09.978467 31021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:09.978753 31021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:09.994093 31021 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.254:43173
I20240503 02:13:09.994112 31086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.254:43173 every 8 connection(s)
I20240503 02:13:09.995330 31021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:13:09.996663 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 31021
I20240503 02:13:09.996874 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:13:09.999554 31087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.011551 31087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5: Bootstrap starting.
I20240503 02:13:10.014003 31087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.014900 31087 log.cc:824] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:10.017160 31087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5: No bootstrap required, opened a new log
I20240503 02:13:10.022874 31087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ac8e5f9d824ced889084d36419cdf5" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 43173 } }
I20240503 02:13:10.023154 31087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.023231 31087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ac8e5f9d824ced889084d36419cdf5, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.023754 31087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ac8e5f9d824ced889084d36419cdf5" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 43173 } }
I20240503 02:13:10.023975 31087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:13:10.024070 31087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:13:10.024194 31087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:10.026043 31087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ac8e5f9d824ced889084d36419cdf5" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 43173 } }
I20240503 02:13:10.026526 31087 leader_election.cc:302] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32ac8e5f9d824ced889084d36419cdf5; no voters:
I20240503 02:13:10.027117 31087 leader_election.cc:288] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:13:10.027210 31092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:10.027832 31092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [term 1 LEADER]: Becoming Leader. State: Replica: 32ac8e5f9d824ced889084d36419cdf5, State: Running, Role: LEADER
I20240503 02:13:10.028350 31092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ac8e5f9d824ced889084d36419cdf5" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 43173 } }
I20240503 02:13:10.028867 31087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:13:10.030776 31094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32ac8e5f9d824ced889084d36419cdf5. Latest consensus state: current_term: 1 leader_uuid: "32ac8e5f9d824ced889084d36419cdf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ac8e5f9d824ced889084d36419cdf5" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 43173 } } }
I20240503 02:13:10.030799 31093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32ac8e5f9d824ced889084d36419cdf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ac8e5f9d824ced889084d36419cdf5" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 43173 } } }
I20240503 02:13:10.031193 31093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:10.031836 31094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5 [sys.catalog]: This master's current role is: LEADER
I20240503 02:13:10.032948 31097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:13:10.038276 31097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:13:10.045722 31097 catalog_manager.cc:1347] Generated new cluster ID: dc52485dd3914114a4eaf66def9142a2
I20240503 02:13:10.045826 31097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:13:10.076710 31097 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:13:10.077862 31097 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:13:10.097970 31097 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 32ac8e5f9d824ced889084d36419cdf5: Generated new TSK 0
I20240503 02:13:10.098753 31097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:13:10.105998 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.193:0
--local_ip_for_outbound_sockets=127.23.104.193
--webserver_interface=127.23.104.193
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43173
--builtin_ntp_servers=127.23.104.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 02:13:10.427248 31111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:10.427515 31111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:10.427613 31111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:10.435070 31111 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:10.435190 31111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:10.435360 31111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.193
I20240503 02:13:10.444046 31111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.23.104.193
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.23.104.254:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.193
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:10.445240 31111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:10.446939 31111 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:10.458586 31116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:10.459301 31117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:10.459928 31119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:10.460615 31111 server_base.cc:1034] running on GCE node
I20240503 02:13:10.461179 31111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:10.462065 31111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:10.463253 31111 hybrid_clock.cc:648] HybridClock initialized: now 1714702390463202 us; error 48 us; skew 500 ppm
I20240503 02:13:10.463548 31111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:10.465687 31111 webserver.cc:460] Webserver started at http://127.23.104.193:37333/ using document root <none> and password file <none>
I20240503 02:13:10.466347 31111 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:10.466487 31111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:10.466795 31111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:10.469269 31111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
format_stamp: "Formatted at 2024-05-03 02:13:10 on dist-test-slave-38ht"
I20240503 02:13:10.469941 31111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
format_stamp: "Formatted at 2024-05-03 02:13:10 on dist-test-slave-38ht"
I20240503 02:13:10.474520 31111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240503 02:13:10.477592 31125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:10.478766 31111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:10.478919 31111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
format_stamp: "Formatted at 2024-05-03 02:13:10 on dist-test-slave-38ht"
I20240503 02:13:10.479099 31111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:10.499363 31111 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:10.500108 31111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:10.500366 31111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:10.501312 31111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:10.502636 31111 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:10.502732 31111 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:10.502847 31111 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:10.502919 31111 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:10.533567 31111 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.193:44019
I20240503 02:13:10.533624 31237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.193:44019 every 8 connection(s)
I20240503 02:13:10.534832 31111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:13:10.537927 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 31111
I20240503 02:13:10.538158 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:13:10.541496 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.194:0
--local_ip_for_outbound_sockets=127.23.104.194
--webserver_interface=127.23.104.194
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43173
--builtin_ntp_servers=127.23.104.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:13:10.547055 31238 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43173
I20240503 02:13:10.547308 31238 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:10.548094 31238 heartbeater.cc:511] Master 127.23.104.254:43173 requested a full tablet report, sending...
I20240503 02:13:10.549938 31052 ts_manager.cc:194] Registered new tserver with Master: b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:10.551324 31052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.193:40711
W20240503 02:13:10.864794 31242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:10.865056 31242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:10.865207 31242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:10.872244 31242 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:10.872355 31242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:10.872519 31242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.194
I20240503 02:13:10.880807 31242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.23.104.194
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.23.104.254:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.194
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:10.881909 31242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:10.883550 31242 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:10.895073 31247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:10.895535 31248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:10.896236 31250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:10.897011 31242 server_base.cc:1034] running on GCE node
I20240503 02:13:10.897528 31242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:10.898342 31242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:10.899521 31242 hybrid_clock.cc:648] HybridClock initialized: now 1714702390899473 us; error 40 us; skew 500 ppm
I20240503 02:13:10.899860 31242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:10.901937 31242 webserver.cc:460] Webserver started at http://127.23.104.194:39431/ using document root <none> and password file <none>
I20240503 02:13:10.902601 31242 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:10.902709 31242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:10.903007 31242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:10.905505 31242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
format_stamp: "Formatted at 2024-05-03 02:13:10 on dist-test-slave-38ht"
I20240503 02:13:10.906200 31242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
format_stamp: "Formatted at 2024-05-03 02:13:10 on dist-test-slave-38ht"
I20240503 02:13:10.910908 31242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240503 02:13:10.913897 31256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:10.915076 31242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:10.915213 31242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
format_stamp: "Formatted at 2024-05-03 02:13:10 on dist-test-slave-38ht"
I20240503 02:13:10.915393 31242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:10.937741 31242 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:10.938524 31242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:10.938766 31242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:10.939623 31242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:10.940981 31242 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:10.941076 31242 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:10.941213 31242 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:10.941272 31242 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:10.972522 31242 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.194:46179
I20240503 02:13:10.972605 31368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.194:46179 every 8 connection(s)
I20240503 02:13:10.973767 31242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:13:10.976845 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 31242
I20240503 02:13:10.977030 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:13:10.980430 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.195:0
--local_ip_for_outbound_sockets=127.23.104.195
--webserver_interface=127.23.104.195
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43173
--builtin_ntp_servers=127.23.104.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:13:10.986827 31369 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43173
I20240503 02:13:10.987093 31369 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:10.987874 31369 heartbeater.cc:511] Master 127.23.104.254:43173 requested a full tablet report, sending...
I20240503 02:13:10.989104 31052 ts_manager.cc:194] Registered new tserver with Master: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:10.989794 31052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.194:52173
W20240503 02:13:11.305359 31372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:11.305621 31372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:11.305721 31372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:11.312500 31372 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:11.312595 31372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:11.312741 31372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.195
I20240503 02:13:11.321018 31372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.23.104.195
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.23.104.254:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.195
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:11.322157 31372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:11.323844 31372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:11.335269 31377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:11.335892 31380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:11.336508 31378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:11.336988 31372 server_base.cc:1034] running on GCE node
I20240503 02:13:11.337477 31372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:11.338290 31372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:11.339468 31372 hybrid_clock.cc:648] HybridClock initialized: now 1714702391339427 us; error 34 us; skew 500 ppm
I20240503 02:13:11.339751 31372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:11.341951 31372 webserver.cc:460] Webserver started at http://127.23.104.195:37747/ using document root <none> and password file <none>
I20240503 02:13:11.342628 31372 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:11.342748 31372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:11.343040 31372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:11.345521 31372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fbb7d16e67d94775b2881712419c0577"
format_stamp: "Formatted at 2024-05-03 02:13:11 on dist-test-slave-38ht"
I20240503 02:13:11.346210 31372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fbb7d16e67d94775b2881712419c0577"
format_stamp: "Formatted at 2024-05-03 02:13:11 on dist-test-slave-38ht"
I20240503 02:13:11.350904 31372 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20240503 02:13:11.353866 31386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:11.354991 31372 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:11.355127 31372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fbb7d16e67d94775b2881712419c0577"
format_stamp: "Formatted at 2024-05-03 02:13:11 on dist-test-slave-38ht"
I20240503 02:13:11.355298 31372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:11.367936 31372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:11.368623 31372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:11.368872 31372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:11.369668 31372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:11.370982 31372 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:11.371066 31372 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:11.371174 31372 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:11.371244 31372 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:11.400708 31372 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.195:39101
I20240503 02:13:11.400751 31498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.195:39101 every 8 connection(s)
I20240503 02:13:11.401809 31372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:13:11.404301 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 31372
I20240503 02:13:11.404536 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:13:11.407241 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.196:0
--local_ip_for_outbound_sockets=127.23.104.196
--webserver_interface=127.23.104.196
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43173
--builtin_ntp_servers=127.23.104.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:13:11.413009 31499 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43173
I20240503 02:13:11.413306 31499 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:11.414258 31499 heartbeater.cc:511] Master 127.23.104.254:43173 requested a full tablet report, sending...
I20240503 02:13:11.415699 31052 ts_manager.cc:194] Registered new tserver with Master: fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:11.416450 31052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.195:56303
I20240503 02:13:11.553926 31238 heartbeater.cc:503] Master 127.23.104.254:43173 was elected leader, sending a full tablet report...
W20240503 02:13:11.736979 31502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:11.737226 31502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:11.737382 31502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:11.744632 31502 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:11.744737 31502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:11.744897 31502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.196
I20240503 02:13:11.753489 31502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.23.104.196
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.23.104.254:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.196
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:11.754660 31502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:11.756520 31502 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:11.768051 31507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:11.768678 31510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:11.769345 31508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:11.769905 31502 server_base.cc:1034] running on GCE node
I20240503 02:13:11.770432 31502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:11.771257 31502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:11.772459 31502 hybrid_clock.cc:648] HybridClock initialized: now 1714702391772428 us; error 23 us; skew 500 ppm
I20240503 02:13:11.772765 31502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:11.774791 31502 webserver.cc:460] Webserver started at http://127.23.104.196:45213/ using document root <none> and password file <none>
I20240503 02:13:11.775458 31502 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:11.775568 31502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:11.775910 31502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:11.778249 31502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b92ff49cec804b1f801c11b64e3259b3"
format_stamp: "Formatted at 2024-05-03 02:13:11 on dist-test-slave-38ht"
I20240503 02:13:11.778949 31502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b92ff49cec804b1f801c11b64e3259b3"
format_stamp: "Formatted at 2024-05-03 02:13:11 on dist-test-slave-38ht"
I20240503 02:13:11.783573 31502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240503 02:13:11.786671 31516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:11.787935 31502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:11.788075 31502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b92ff49cec804b1f801c11b64e3259b3"
format_stamp: "Formatted at 2024-05-03 02:13:11 on dist-test-slave-38ht"
I20240503 02:13:11.788251 31502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:11.801183 31502 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:11.801882 31502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:11.802124 31502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:11.802947 31502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:11.804293 31502 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:11.804391 31502 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:11.804503 31502 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:11.804574 31502 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:11.835182 31502 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.196:39849
I20240503 02:13:11.835232 31628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.196:39849 every 8 connection(s)
I20240503 02:13:11.836462 31502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:13:11.846190 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 31502
I20240503 02:13:11.846405 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:13:11.848018 31629 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43173
I20240503 02:13:11.848313 31629 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:11.849023 31629 heartbeater.cc:511] Master 127.23.104.254:43173 requested a full tablet report, sending...
I20240503 02:13:11.849318 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.197:0
--local_ip_for_outbound_sockets=127.23.104.197
--webserver_interface=127.23.104.197
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43173
--builtin_ntp_servers=127.23.104.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:13:11.850292 31052 ts_manager.cc:194] Registered new tserver with Master: b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:11.851006 31052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.196:37293
I20240503 02:13:11.992177 31369 heartbeater.cc:503] Master 127.23.104.254:43173 was elected leader, sending a full tablet report...
W20240503 02:13:12.167335 31632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:12.167603 31632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:12.167688 31632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:12.174242 31632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:12.174340 31632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:12.174455 31632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.197
I20240503 02:13:12.182488 31632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.23.104.197
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.23.104.254:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.197
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:12.183677 31632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:12.185297 31632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:12.196360 31638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:12.197059 31640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:12.197497 31637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:12.198154 31632 server_base.cc:1034] running on GCE node
I20240503 02:13:12.198736 31632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:12.199628 31632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:12.200824 31632 hybrid_clock.cc:648] HybridClock initialized: now 1714702392200781 us; error 50 us; skew 500 ppm
I20240503 02:13:12.201103 31632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:12.206874 31632 webserver.cc:460] Webserver started at http://127.23.104.197:33139/ using document root <none> and password file <none>
I20240503 02:13:12.207584 31632 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:12.207708 31632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:12.208052 31632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:12.210563 31632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "13d0f62249b34b01a3a47c6afe10e750"
format_stamp: "Formatted at 2024-05-03 02:13:12 on dist-test-slave-38ht"
I20240503 02:13:12.211273 31632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "13d0f62249b34b01a3a47c6afe10e750"
format_stamp: "Formatted at 2024-05-03 02:13:12 on dist-test-slave-38ht"
I20240503 02:13:12.215958 31632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.005s
I20240503 02:13:12.218878 31646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:12.220101 31632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240503 02:13:12.220244 31632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "13d0f62249b34b01a3a47c6afe10e750"
format_stamp: "Formatted at 2024-05-03 02:13:12 on dist-test-slave-38ht"
I20240503 02:13:12.220412 31632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:12.237123 31632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:12.237851 31632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:12.238088 31632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:12.238963 31632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:12.240331 31632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:12.240442 31632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:12.240551 31632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:12.240629 31632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:12.271323 31632 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.197:37543
I20240503 02:13:12.271373 31758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.197:37543 every 8 connection(s)
I20240503 02:13:12.272547 31632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:13:12.276590 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 31632
I20240503 02:13:12.276743 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:13:12.279570 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.198:0
--local_ip_for_outbound_sockets=127.23.104.198
--webserver_interface=127.23.104.198
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43173
--builtin_ntp_servers=127.23.104.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:13:12.284442 31759 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43173
I20240503 02:13:12.284734 31759 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:12.285475 31759 heartbeater.cc:511] Master 127.23.104.254:43173 requested a full tablet report, sending...
I20240503 02:13:12.286857 31052 ts_manager.cc:194] Registered new tserver with Master: 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543)
I20240503 02:13:12.287930 31052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.197:60153
I20240503 02:13:12.419214 31499 heartbeater.cc:503] Master 127.23.104.254:43173 was elected leader, sending a full tablet report...
W20240503 02:13:12.606173 31762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:12.606441 31762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:12.606593 31762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:12.613852 31762 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:12.613974 31762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:12.614125 31762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.198
I20240503 02:13:12.622592 31762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.23.104.198
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.23.104.254:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.198
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:12.623716 31762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:12.625375 31762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:12.637087 31770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:12.637687 31768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:12.638435 31767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:12.639259 31762 server_base.cc:1034] running on GCE node
I20240503 02:13:12.639861 31762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:12.640699 31762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:12.641876 31762 hybrid_clock.cc:648] HybridClock initialized: now 1714702392641836 us; error 32 us; skew 500 ppm
I20240503 02:13:12.642145 31762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:12.648602 31762 webserver.cc:460] Webserver started at http://127.23.104.198:34573/ using document root <none> and password file <none>
I20240503 02:13:12.649262 31762 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:12.649361 31762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:12.649654 31762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:12.652065 31762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "05ef0a15decc468ea3a1af8828b0f630"
format_stamp: "Formatted at 2024-05-03 02:13:12 on dist-test-slave-38ht"
I20240503 02:13:12.652774 31762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "05ef0a15decc468ea3a1af8828b0f630"
format_stamp: "Formatted at 2024-05-03 02:13:12 on dist-test-slave-38ht"
I20240503 02:13:12.657402 31762 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:13:12.660820 31776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:12.661923 31762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:12.662071 31762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "05ef0a15decc468ea3a1af8828b0f630"
format_stamp: "Formatted at 2024-05-03 02:13:12 on dist-test-slave-38ht"
I20240503 02:13:12.662235 31762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:12.682437 31762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:12.683156 31762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:12.683398 31762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:12.684316 31762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:12.685642 31762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:12.685724 31762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:12.685819 31762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:12.685870 31762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:12.715380 31762 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.198:38195
I20240503 02:13:12.715445 31888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.198:38195 every 8 connection(s)
I20240503 02:13:12.716575 31762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:13:12.719130 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 31762
I20240503 02:13:12.719396 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:13:12.722663 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.199:0
--local_ip_for_outbound_sockets=127.23.104.199
--webserver_interface=127.23.104.199
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43173
--builtin_ntp_servers=127.23.104.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:13:12.728804 31889 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43173
I20240503 02:13:12.729072 31889 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:12.729811 31889 heartbeater.cc:511] Master 127.23.104.254:43173 requested a full tablet report, sending...
I20240503 02:13:12.731014 31052 ts_manager.cc:194] Registered new tserver with Master: 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195)
I20240503 02:13:12.731720 31052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.198:53403
I20240503 02:13:12.853160 31629 heartbeater.cc:503] Master 127.23.104.254:43173 was elected leader, sending a full tablet report...
W20240503 02:13:13.042364 31892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:13.042637 31892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:13.042788 31892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:13.049763 31892 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:13.049890 31892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:13.050040 31892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.199
I20240503 02:13:13.058488 31892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.23.104.199
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.23.104.254:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.199
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:13.059715 31892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:13.061394 31892 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:13.073612 31898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:13.074193 31900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:13.074668 31897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:13.075325 31892 server_base.cc:1034] running on GCE node
I20240503 02:13:13.075876 31892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:13.076676 31892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:13.077883 31892 hybrid_clock.cc:648] HybridClock initialized: now 1714702393077830 us; error 33 us; skew 500 ppm
I20240503 02:13:13.078263 31892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:13.080500 31892 webserver.cc:460] Webserver started at http://127.23.104.199:35583/ using document root <none> and password file <none>
I20240503 02:13:13.081189 31892 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:13.081287 31892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:13.081544 31892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:13.084115 31892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "4a3e3937b61447598fd76bc57f5d0939"
format_stamp: "Formatted at 2024-05-03 02:13:13 on dist-test-slave-38ht"
I20240503 02:13:13.084795 31892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "4a3e3937b61447598fd76bc57f5d0939"
format_stamp: "Formatted at 2024-05-03 02:13:13 on dist-test-slave-38ht"
I20240503 02:13:13.089452 31892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:13:13.092629 31906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:13.093856 31892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:13.093979 31892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4a3e3937b61447598fd76bc57f5d0939"
format_stamp: "Formatted at 2024-05-03 02:13:13 on dist-test-slave-38ht"
I20240503 02:13:13.094153 31892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:13.106765 31892 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:13.107416 31892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:13.107632 31892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:13.108482 31892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:13.109815 31892 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:13.109899 31892 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:13.109974 31892 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:13.110026 31892 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:13.141062 31892 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.199:36997
I20240503 02:13:13.141127 32018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.199:36997 every 8 connection(s)
I20240503 02:13:13.142306 31892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:13:13.152060 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 31892
I20240503 02:13:13.152261 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:13:13.154140 32019 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43173
I20240503 02:13:13.154405 32019 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:13.155220 32019 heartbeater.cc:511] Master 127.23.104.254:43173 requested a full tablet report, sending...
I20240503 02:13:13.155531 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.200:0
--local_ip_for_outbound_sockets=127.23.104.200
--webserver_interface=127.23.104.200
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43173
--builtin_ntp_servers=127.23.104.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:13:13.156607 31051 ts_manager.cc:194] Registered new tserver with Master: 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997)
I20240503 02:13:13.157810 31051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.199:44349
I20240503 02:13:13.290186 31759 heartbeater.cc:503] Master 127.23.104.254:43173 was elected leader, sending a full tablet report...
W20240503 02:13:13.475945 32022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:13.476187 32022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:13.476292 32022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:13.483204 32022 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:13.483295 32022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:13.483417 32022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.200
I20240503 02:13:13.491484 32022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.23.104.200
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.23.104.254:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.200
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:13.492610 32022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:13.494163 32022 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:13.505290 32027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:13.506055 32028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:13.506685 32030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:13.507510 32022 server_base.cc:1034] running on GCE node
I20240503 02:13:13.508075 32022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:13.508860 32022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:13.510042 32022 hybrid_clock.cc:648] HybridClock initialized: now 1714702393510003 us; error 36 us; skew 500 ppm
I20240503 02:13:13.510314 32022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:13.512378 32022 webserver.cc:460] Webserver started at http://127.23.104.200:37389/ using document root <none> and password file <none>
I20240503 02:13:13.513029 32022 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:13.513149 32022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:13.513446 32022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:13.515849 32022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "0429a57df6ba4f92a73e779b04b05040"
format_stamp: "Formatted at 2024-05-03 02:13:13 on dist-test-slave-38ht"
I20240503 02:13:13.516571 32022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "0429a57df6ba4f92a73e779b04b05040"
format_stamp: "Formatted at 2024-05-03 02:13:13 on dist-test-slave-38ht"
I20240503 02:13:13.521277 32022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:13:13.524026 32036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:13.525233 32022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:13.525379 32022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0429a57df6ba4f92a73e779b04b05040"
format_stamp: "Formatted at 2024-05-03 02:13:13 on dist-test-slave-38ht"
I20240503 02:13:13.525552 32022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:13.557660 32022 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:13.558501 32022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:13.558817 32022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:13.559929 32022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:13.561249 32022 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:13.561329 32022 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:13.561401 32022 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:13.561455 32022 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:13.590539 32022 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.200:37401
I20240503 02:13:13.590624 32148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.200:37401 every 8 connection(s)
I20240503 02:13:13.591733 32022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:13:13.594599 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 32022
I20240503 02:13:13.594825 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:13:13.604210 32149 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43173
I20240503 02:13:13.604472 32149 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:13.605201 32149 heartbeater.cc:511] Master 127.23.104.254:43173 requested a full tablet report, sending...
I20240503 02:13:13.606457 31051 ts_manager.cc:194] Registered new tserver with Master: 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401)
I20240503 02:13:13.607208 31051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.200:43733
I20240503 02:13:13.610956 23971 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:13:13.625679 23971 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:13:13.625802 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 31632
I20240503 02:13:13.633015 31754 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:13:13.710948 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 31632
I20240503 02:13:13.726732 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 31762
I20240503 02:13:13.734010 31889 heartbeater.cc:503] Master 127.23.104.254:43173 was elected leader, sending a full tablet report...
I20240503 02:13:13.734607 31884 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:13:13.811143 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 31762
I20240503 02:13:13.826434 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 31892
I20240503 02:13:13.834125 32014 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:13:13.911187 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 31892
I20240503 02:13:13.926242 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 32022
I20240503 02:13:13.934208 32144 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:13:14.016287 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 32022
I20240503 02:13:17.783576 31051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39754:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:17.831321 31563 tablet_service.cc:1467] Processing CreateTablet for tablet 80a15c3541fc4068acac595bd909f8d8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=cc3c62c772ec4610892f67e8c6d52379]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.832428 31564 tablet_service.cc:1467] Processing CreateTablet for tablet 00e1dad5efda4e9eae55a8f9d80f0f96 (DEFAULT_TABLE table=rebalance_test_table_0 [id=cc3c62c772ec4610892f67e8c6d52379]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.832079 31434 tablet_service.cc:1467] Processing CreateTablet for tablet 00e1dad5efda4e9eae55a8f9d80f0f96 (DEFAULT_TABLE table=rebalance_test_table_0 [id=cc3c62c772ec4610892f67e8c6d52379]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.832033 31433 tablet_service.cc:1467] Processing CreateTablet for tablet 37ca6554fec24a07865da3eabef27152 (DEFAULT_TABLE table=rebalance_test_table_0 [id=cc3c62c772ec4610892f67e8c6d52379]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.834234 31563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80a15c3541fc4068acac595bd909f8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.835351 31434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e1dad5efda4e9eae55a8f9d80f0f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.835767 31564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e1dad5efda4e9eae55a8f9d80f0f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.839066 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ca6554fec24a07865da3eabef27152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.839237 31171 tablet_service.cc:1467] Processing CreateTablet for tablet 80a15c3541fc4068acac595bd909f8d8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=cc3c62c772ec4610892f67e8c6d52379]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.840423 31171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80a15c3541fc4068acac595bd909f8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.841346 31304 tablet_service.cc:1467] Processing CreateTablet for tablet 37ca6554fec24a07865da3eabef27152 (DEFAULT_TABLE table=rebalance_test_table_0 [id=cc3c62c772ec4610892f67e8c6d52379]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.842204 31172 tablet_service.cc:1467] Processing CreateTablet for tablet 37ca6554fec24a07865da3eabef27152 (DEFAULT_TABLE table=rebalance_test_table_0 [id=cc3c62c772ec4610892f67e8c6d52379]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.842638 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ca6554fec24a07865da3eabef27152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.842697 31172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ca6554fec24a07865da3eabef27152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.835592 31173 tablet_service.cc:1467] Processing CreateTablet for tablet 00e1dad5efda4e9eae55a8f9d80f0f96 (DEFAULT_TABLE table=rebalance_test_table_0 [id=cc3c62c772ec4610892f67e8c6d52379]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.843667 31303 tablet_service.cc:1467] Processing CreateTablet for tablet 80a15c3541fc4068acac595bd909f8d8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=cc3c62c772ec4610892f67e8c6d52379]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.843809 31173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e1dad5efda4e9eae55a8f9d80f0f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.844081 31303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80a15c3541fc4068acac595bd909f8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.854972 32178 tablet_bootstrap.cc:492] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:17.858744 32178 tablet_bootstrap.cc:654] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.860587 32178 log.cc:824] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:17.868749 32180 tablet_bootstrap.cc:492] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:17.875447 32180 tablet_bootstrap.cc:654] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.876936 32178 tablet_bootstrap.cc:492] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:17.877205 32178 ts_tablet_manager.cc:1367] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.010s
I20240503 02:13:17.878386 32180 log.cc:824] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:17.888739 32178 raft_consensus.cc:348] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:17.889292 32178 raft_consensus.cc:374] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.889384 32178 raft_consensus.cc:729] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.890574 32178 consensus_queue.cc:260] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:17.891439 32179 tablet_bootstrap.cc:492] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:17.892616 32178 ts_tablet_manager.cc:1398] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20240503 02:13:17.891434 32183 tablet_bootstrap.cc:492] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:17.893003 32178 tablet_bootstrap.cc:492] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:17.894548 32178 tablet_bootstrap.cc:654] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.894738 32179 tablet_bootstrap.cc:654] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.895936 32179 log.cc:824] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:17.896100 32183 tablet_bootstrap.cc:654] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.897397 32183 log.cc:824] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:17.901278 32180 tablet_bootstrap.cc:492] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:17.901641 32180 ts_tablet_manager.cc:1367] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.009s
I20240503 02:13:17.902496 32178 tablet_bootstrap.cc:492] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:17.902647 32178 ts_tablet_manager.cc:1367] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240503 02:13:17.903578 32178 raft_consensus.cc:348] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:17.903818 32178 raft_consensus.cc:374] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.903923 32178 raft_consensus.cc:729] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.904147 32178 consensus_queue.cc:260] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:17.904752 32178 ts_tablet_manager.cc:1398] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:17.907248 32179 tablet_bootstrap.cc:492] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:17.907516 32179 ts_tablet_manager.cc:1367] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20240503 02:13:17.907545 32183 tablet_bootstrap.cc:492] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:17.907866 32183 ts_tablet_manager.cc:1367] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20240503 02:13:17.908389 32184 raft_consensus.cc:482] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:17.908592 32184 raft_consensus.cc:504] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:17.910346 32184 leader_election.cc:288] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:17.912030 32180 raft_consensus.cc:348] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:17.912536 32180 raft_consensus.cc:374] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.912672 32180 raft_consensus.cc:729] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.913537 32180 consensus_queue.cc:260] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:17.916541 32179 raft_consensus.cc:348] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:17.916983 32179 raft_consensus.cc:374] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.917098 32179 raft_consensus.cc:729] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.917274 32183 raft_consensus.cc:348] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:17.917719 32183 raft_consensus.cc:374] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.917857 32183 raft_consensus.cc:729] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.917863 32179 consensus_queue.cc:260] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:17.918774 32183 consensus_queue.cc:260] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:17.928006 32179 ts_tablet_manager.cc:1398] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.020s user 0.013s sys 0.009s
I20240503 02:13:17.928551 32179 tablet_bootstrap.cc:492] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:17.930864 32179 tablet_bootstrap.cc:654] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.939617 32179 tablet_bootstrap.cc:492] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:17.939860 32179 ts_tablet_manager.cc:1367] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240503 02:13:17.940873 32179 raft_consensus.cc:348] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:17.941174 32179 raft_consensus.cc:374] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.941325 32179 raft_consensus.cc:729] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.941578 32179 consensus_queue.cc:260] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:17.942174 32179 ts_tablet_manager.cc:1398] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:17.942504 32179 tablet_bootstrap.cc:492] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:17.946703 32179 tablet_bootstrap.cc:654] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.947616 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80a15c3541fc4068acac595bd909f8d8" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:17.948140 31193 raft_consensus.cc:2454] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 0.
I20240503 02:13:17.948675 32183 ts_tablet_manager.cc:1398] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.041s user 0.019s sys 0.008s
I20240503 02:13:17.949167 32183 tablet_bootstrap.cc:492] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:17.950007 32180 ts_tablet_manager.cc:1398] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.048s user 0.027s sys 0.017s
I20240503 02:13:17.950368 32180 tablet_bootstrap.cc:492] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:17.951257 32183 tablet_bootstrap.cc:654] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.951917 32180 tablet_bootstrap.cc:654] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.954972 31519 leader_election.cc:302] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, b92ff49cec804b1f801c11b64e3259b3; no voters:
I20240503 02:13:17.955276 32180 tablet_bootstrap.cc:492] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:17.955418 32180 ts_tablet_manager.cc:1367] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 02:13:17.955392 32184 raft_consensus.cc:2781] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:17.955531 32184 raft_consensus.cc:482] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:17.955657 32184 raft_consensus.cc:3037] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.956430 32180 raft_consensus.cc:348] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:17.956616 32180 raft_consensus.cc:374] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.956699 32180 raft_consensus.cc:729] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.956882 32180 consensus_queue.cc:260] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:17.957437 32180 ts_tablet_manager.cc:1398] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:17.958487 32184 raft_consensus.cc:504] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:17.958832 32179 tablet_bootstrap.cc:492] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:17.958990 32179 ts_tablet_manager.cc:1367] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.008s
I20240503 02:13:17.959317 32184 leader_election.cc:288] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Requested vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:17.959812 32179 raft_consensus.cc:348] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:17.960038 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80a15c3541fc4068acac595bd909f8d8" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:17.960125 32179 raft_consensus.cc:374] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.960196 32179 raft_consensus.cc:729] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.960223 31193 raft_consensus.cc:3037] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.960458 32179 consensus_queue.cc:260] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:17.961042 32179 ts_tablet_manager.cc:1398] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:17.962491 32183 tablet_bootstrap.cc:492] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:17.962661 32183 ts_tablet_manager.cc:1367] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20240503 02:13:17.963251 31193 raft_consensus.cc:2454] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 1.
I20240503 02:13:17.963699 32192 raft_consensus.cc:482] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:17.963871 32183 raft_consensus.cc:348] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:17.964098 32183 raft_consensus.cc:374] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.964004 31519 leader_election.cc:302] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, b92ff49cec804b1f801c11b64e3259b3; no voters:
I20240503 02:13:17.963987 32192 raft_consensus.cc:504] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:17.964212 32183 raft_consensus.cc:729] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.964571 32184 raft_consensus.cc:2781] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:17.964773 32183 consensus_queue.cc:260] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:17.965812 32184 raft_consensus.cc:686] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Becoming Leader. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Running, Role: LEADER
I20240503 02:13:17.966257 32192 leader_election.cc:288] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:17.966483 32183 ts_tablet_manager.cc:1398] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:13:17.966341 32184 consensus_queue.cc:237] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:17.967656 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80a15c3541fc4068acac595bd909f8d8" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
I20240503 02:13:17.967753 31324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80a15c3541fc4068acac595bd909f8d8" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" is_pre_election: true
I20240503 02:13:17.968124 31323 raft_consensus.cc:3037] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.972748 31323 raft_consensus.cc:2454] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 1.
W20240503 02:13:17.980640 31370 tablet.cc:2369] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:13:17.988154 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ca6554fec24a07865da3eabef27152" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" is_pre_election: true
I20240503 02:13:17.988356 31323 raft_consensus.cc:2454] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 0.
I20240503 02:13:17.988828 31388 leader_election.cc:302] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:17.989358 32198 raft_consensus.cc:2781] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:17.989528 32198 raft_consensus.cc:482] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:17.989904 32198 raft_consensus.cc:3037] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.991914 31052 catalog_manager.cc:5484] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196). New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:17.992980 32198 raft_consensus.cc:504] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:17.993625 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ca6554fec24a07865da3eabef27152" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:17.993857 31193 raft_consensus.cc:2454] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 0.
I20240503 02:13:17.994274 32198 leader_election.cc:288] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 election: Requested vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:17.994354 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ca6554fec24a07865da3eabef27152" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
I20240503 02:13:17.994539 31323 raft_consensus.cc:3037] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.994818 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ca6554fec24a07865da3eabef27152" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:17.995007 31193 raft_consensus.cc:3037] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.997002 31323 raft_consensus.cc:2454] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 1.
I20240503 02:13:17.997320 31388 leader_election.cc:302] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:17.997526 32198 raft_consensus.cc:2781] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:17.997538 31193 raft_consensus.cc:2454] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 1.
I20240503 02:13:17.997701 32198 raft_consensus.cc:686] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Becoming Leader. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Running, Role: LEADER
I20240503 02:13:17.998109 32198 consensus_queue.cc:237] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:18.002034 31052 catalog_manager.cc:5484] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbb7d16e67d94775b2881712419c0577 (127.23.104.195). New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } }
W20240503 02:13:18.042081 31239 tablet.cc:2369] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:13:18.044931 32184 raft_consensus.cc:482] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:18.045154 32184 raft_consensus.cc:504] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:18.045807 32184 leader_election.cc:288] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:18.046299 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e1dad5efda4e9eae55a8f9d80f0f96" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:18.046517 31193 raft_consensus.cc:2454] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 0.
I20240503 02:13:18.046903 31519 leader_election.cc:302] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, b92ff49cec804b1f801c11b64e3259b3; no voters:
I20240503 02:13:18.047221 32184 raft_consensus.cc:2781] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:18.047309 32184 raft_consensus.cc:482] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:18.047415 32184 raft_consensus.cc:3037] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.049392 32184 raft_consensus.cc:504] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:18.049945 32184 leader_election.cc:288] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Requested vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:18.050335 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e1dad5efda4e9eae55a8f9d80f0f96" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:18.050521 31193 raft_consensus.cc:3037] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.052084 31453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e1dad5efda4e9eae55a8f9d80f0f96" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:18.052115 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e1dad5efda4e9eae55a8f9d80f0f96" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:18.052484 31454 raft_consensus.cc:2454] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 0.
I20240503 02:13:18.053004 31193 raft_consensus.cc:2454] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 1.
I20240503 02:13:18.053313 31519 leader_election.cc:302] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, b92ff49cec804b1f801c11b64e3259b3; no voters: fbb7d16e67d94775b2881712419c0577
I20240503 02:13:18.053589 32184 raft_consensus.cc:2781] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:18.053761 32184 raft_consensus.cc:686] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Becoming Leader. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Running, Role: LEADER
I20240503 02:13:18.054016 32184 consensus_queue.cc:237] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:18.056525 31052 catalog_manager.cc:5484] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196). New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:18.092290 31630 tablet.cc:2369] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:13:18.157814 31500 tablet.cc:2369] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:13:18.436071 32204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:18.436403 32204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:18.443673 32204 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:18.453716 32184 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:18.466147 32184 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:18.489058 32184 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:18.491089 32198 consensus_queue.cc:1035] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:18.493654 31453 raft_consensus.cc:3037] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.497284 32197 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 02:13:18.510433 32198 consensus_queue.cc:1035] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:18.759940 31518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:46179, user_credentials={real_user=slave}} blocked reactor thread for 63784.3us
INSERT report
rows total: 2000
time total: 295.878 ms
time per row: 0.147939 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:13:19.860296 31049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39754:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:19.868678 31563 tablet_service.cc:1467] Processing CreateTablet for tablet 9e964b80f5414d9ea3a5aea07b99d1d4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d3a62fd4d46c4ba082abbec8f97c84e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.868853 31304 tablet_service.cc:1467] Processing CreateTablet for tablet 9e964b80f5414d9ea3a5aea07b99d1d4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d3a62fd4d46c4ba082abbec8f97c84e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.869153 31563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e964b80f5414d9ea3a5aea07b99d1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.869305 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e964b80f5414d9ea3a5aea07b99d1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.869700 31433 tablet_service.cc:1467] Processing CreateTablet for tablet 9e964b80f5414d9ea3a5aea07b99d1d4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d3a62fd4d46c4ba082abbec8f97c84e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.870206 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e964b80f5414d9ea3a5aea07b99d1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.870034 31434 tablet_service.cc:1467] Processing CreateTablet for tablet c185751912de4f78808785a620dd9746 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d3a62fd4d46c4ba082abbec8f97c84e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.870503 31434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c185751912de4f78808785a620dd9746. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.876775 31303 tablet_service.cc:1467] Processing CreateTablet for tablet c185751912de4f78808785a620dd9746 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d3a62fd4d46c4ba082abbec8f97c84e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.877126 31303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c185751912de4f78808785a620dd9746. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.877286 31564 tablet_service.cc:1467] Processing CreateTablet for tablet c185751912de4f78808785a620dd9746 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d3a62fd4d46c4ba082abbec8f97c84e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.877760 31564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c185751912de4f78808785a620dd9746. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.883332 31302 tablet_service.cc:1467] Processing CreateTablet for tablet c91a7b003ed9411aba65a46d00c7d95b (DEFAULT_TABLE table=rebalance_test_table_1 [id=d3a62fd4d46c4ba082abbec8f97c84e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.883823 31302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91a7b003ed9411aba65a46d00c7d95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.884272 32277 tablet_bootstrap.cc:492] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:19.884552 31432 tablet_service.cc:1467] Processing CreateTablet for tablet c91a7b003ed9411aba65a46d00c7d95b (DEFAULT_TABLE table=rebalance_test_table_1 [id=d3a62fd4d46c4ba082abbec8f97c84e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.884936 31432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91a7b003ed9411aba65a46d00c7d95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.885618 32276 tablet_bootstrap.cc:492] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:19.886377 32277 tablet_bootstrap.cc:654] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.887181 32276 tablet_bootstrap.cc:654] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.888499 31172 tablet_service.cc:1467] Processing CreateTablet for tablet c91a7b003ed9411aba65a46d00c7d95b (DEFAULT_TABLE table=rebalance_test_table_1 [id=d3a62fd4d46c4ba082abbec8f97c84e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.889031 31172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91a7b003ed9411aba65a46d00c7d95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.892562 32278 tablet_bootstrap.cc:492] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:19.894202 32277 tablet_bootstrap.cc:492] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:19.894212 32278 tablet_bootstrap.cc:654] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.894408 32277 ts_tablet_manager.cc:1367] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240503 02:13:19.895221 32276 tablet_bootstrap.cc:492] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:19.895202 32277 raft_consensus.cc:348] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.895448 32277 raft_consensus.cc:374] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.895563 32276 ts_tablet_manager.cc:1367] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 02:13:19.895577 32277 raft_consensus.cc:729] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.895879 32277 consensus_queue.cc:260] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.896684 32276 raft_consensus.cc:348] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.898334 32276 raft_consensus.cc:374] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.898527 32276 raft_consensus.cc:729] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.898931 32276 consensus_queue.cc:260] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.899933 32276 ts_tablet_manager.cc:1398] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:19.900625 32282 tablet_bootstrap.cc:492] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:19.903502 32282 tablet_bootstrap.cc:654] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.904395 32277 ts_tablet_manager.cc:1398] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20240503 02:13:19.904728 32277 tablet_bootstrap.cc:492] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:19.905706 32276 tablet_bootstrap.cc:492] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:19.906467 32277 tablet_bootstrap.cc:654] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.909044 32276 tablet_bootstrap.cc:654] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.909893 32278 tablet_bootstrap.cc:492] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:19.909922 32277 tablet_bootstrap.cc:492] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:19.910040 32278 ts_tablet_manager.cc:1367] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.009s
I20240503 02:13:19.910054 32277 ts_tablet_manager.cc:1367] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 02:13:19.910746 32276 tablet_bootstrap.cc:492] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:19.910738 32277 raft_consensus.cc:348] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.910789 32278 raft_consensus.cc:348] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.910943 32276 ts_tablet_manager.cc:1367] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:13:19.910984 32278 raft_consensus.cc:374] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.911011 32277 raft_consensus.cc:374] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.911073 32278 raft_consensus.cc:729] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.911096 32277 raft_consensus.cc:729] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.911108 32282 tablet_bootstrap.cc:492] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:19.911242 32282 ts_tablet_manager.cc:1367] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 02:13:19.911326 32277 consensus_queue.cc:260] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.911617 32276 raft_consensus.cc:348] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.911899 32277 ts_tablet_manager.cc:1398] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:13:19.911908 32276 raft_consensus.cc:374] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.911994 32276 raft_consensus.cc:729] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.911952 32282 raft_consensus.cc:348] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.912179 32282 raft_consensus.cc:374] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.912232 32277 tablet_bootstrap.cc:492] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:19.912303 32282 raft_consensus.cc:729] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.912437 32278 consensus_queue.cc:260] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.912551 32282 consensus_queue.cc:260] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.913300 32276 consensus_queue.cc:260] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.914132 32277 tablet_bootstrap.cc:654] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.916366 32278 ts_tablet_manager.cc:1398] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:19.918094 32278 tablet_bootstrap.cc:492] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:19.918404 32276 ts_tablet_manager.cc:1398] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:19.920838 32282 ts_tablet_manager.cc:1398] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20240503 02:13:19.921857 32277 tablet_bootstrap.cc:492] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:19.921983 32278 tablet_bootstrap.cc:654] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.922044 32277 ts_tablet_manager.cc:1367] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20240503 02:13:19.922680 32277 raft_consensus.cc:348] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.922875 32277 raft_consensus.cc:374] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.922988 32277 raft_consensus.cc:729] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.923323 32277 consensus_queue.cc:260] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.923950 32278 tablet_bootstrap.cc:492] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:19.923959 32277 ts_tablet_manager.cc:1398] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:19.924158 32278 ts_tablet_manager.cc:1367] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 02:13:19.924861 32278 raft_consensus.cc:348] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.925103 32278 raft_consensus.cc:374] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.925207 32278 raft_consensus.cc:729] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.925487 32278 consensus_queue.cc:260] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.925935 32278 ts_tablet_manager.cc:1398] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:19.926218 32278 tablet_bootstrap.cc:492] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:19.927690 32278 tablet_bootstrap.cc:654] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.929365 32278 tablet_bootstrap.cc:492] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:19.929484 32278 ts_tablet_manager.cc:1367] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:19.930160 32278 raft_consensus.cc:348] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.930351 32278 raft_consensus.cc:374] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.930445 32278 raft_consensus.cc:729] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.930608 32278 consensus_queue.cc:260] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.930986 32278 ts_tablet_manager.cc:1398] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:13:19.953543 32253 raft_consensus.cc:482] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:19.953682 32253 raft_consensus.cc:504] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.954289 32253 leader_election.cc:288] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:19.954721 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c185751912de4f78808785a620dd9746" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:19.954749 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c185751912de4f78808785a620dd9746" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" is_pre_election: true
I20240503 02:13:19.954916 31454 raft_consensus.cc:2454] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 0.
I20240503 02:13:19.954921 31323 raft_consensus.cc:2454] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 0.
I20240503 02:13:19.955325 31518 leader_election.cc:302] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:19.955566 32253 raft_consensus.cc:2781] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:19.955682 32253 raft_consensus.cc:482] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:19.955797 32253 raft_consensus.cc:3037] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.957643 32253 raft_consensus.cc:504] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.958190 32253 leader_election.cc:288] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Requested vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:19.958696 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c185751912de4f78808785a620dd9746" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
I20240503 02:13:19.958722 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c185751912de4f78808785a620dd9746" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:19.958873 31323 raft_consensus.cc:3037] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.958897 31454 raft_consensus.cc:3037] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.960914 31323 raft_consensus.cc:2454] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 1.
I20240503 02:13:19.960927 31454 raft_consensus.cc:2454] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 1.
I20240503 02:13:19.961326 31517 leader_election.cc:302] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:19.961608 32253 raft_consensus.cc:2781] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:19.961969 32253 raft_consensus.cc:686] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Becoming Leader. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Running, Role: LEADER
I20240503 02:13:19.962242 32253 consensus_queue.cc:237] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.964641 31051 catalog_manager.cc:5484] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196). New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:19.969611 32283 raft_consensus.cc:482] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:19.969751 32283 raft_consensus.cc:504] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.971275 32283 leader_election.cc:288] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:19.990480 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91a7b003ed9411aba65a46d00c7d95b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:19.990717 31192 raft_consensus.cc:2454] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:19.990958 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91a7b003ed9411aba65a46d00c7d95b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:19.991169 31454 raft_consensus.cc:2454] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:19.991182 31259 leader_election.cc:302] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:19.991520 32283 raft_consensus.cc:2781] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:19.991652 32283 raft_consensus.cc:482] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:19.991730 32283 raft_consensus.cc:3037] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.994606 32283 raft_consensus.cc:504] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:19.995258 32283 leader_election.cc:288] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Requested vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:19.995678 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91a7b003ed9411aba65a46d00c7d95b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:19.995865 31192 raft_consensus.cc:3037] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.997866 31192 raft_consensus.cc:2454] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:19.998185 31259 leader_election.cc:302] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:19.998471 32283 raft_consensus.cc:2781] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:19.998466 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91a7b003ed9411aba65a46d00c7d95b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:19.998651 31454 raft_consensus.cc:3037] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.000066 32283 raft_consensus.cc:686] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Leader. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:20.000599 31454 raft_consensus.cc:2454] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:20.000824 32283 consensus_queue.cc:237] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:20.004249 31051 catalog_manager.cc:5484] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:20.038103 32283 raft_consensus.cc:482] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:20.038316 32283 raft_consensus.cc:504] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:20.038982 32283 leader_election.cc:288] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:20.039366 32271 raft_consensus.cc:482] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:20.039544 32271 raft_consensus.cc:504] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:20.039713 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e964b80f5414d9ea3a5aea07b99d1d4" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:20.039991 31454 raft_consensus.cc:2454] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:20.040426 31257 leader_election.cc:302] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:20.040477 32271 leader_election.cc:288] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:20.040766 32283 raft_consensus.cc:2781] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:20.040869 32283 raft_consensus.cc:482] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:20.041020 32283 raft_consensus.cc:3037] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.041512 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e964b80f5414d9ea3a5aea07b99d1d4" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" is_pre_election: true
I20240503 02:13:20.044492 32283 raft_consensus.cc:504] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:20.044958 31323 raft_consensus.cc:2379] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fbb7d16e67d94775b2881712419c0577 in current term 1: Already voted for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in this term.
I20240503 02:13:20.045833 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e964b80f5414d9ea3a5aea07b99d1d4" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:20.046036 31454 raft_consensus.cc:3037] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.046355 32283 leader_election.cc:288] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Requested vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:20.048134 31584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e964b80f5414d9ea3a5aea07b99d1d4" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3" is_pre_election: true
I20240503 02:13:20.048375 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e964b80f5414d9ea3a5aea07b99d1d4" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:20.048564 31584 raft_consensus.cc:2454] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:20.048717 31454 raft_consensus.cc:2454] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:20.049165 31257 leader_election.cc:302] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:20.049425 32283 raft_consensus.cc:2781] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:20.049607 32283 raft_consensus.cc:686] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Leader. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:20.049794 31584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e964b80f5414d9ea3a5aea07b99d1d4" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3" is_pre_election: true
I20240503 02:13:20.049865 32283 consensus_queue.cc:237] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:20.049968 31584 raft_consensus.cc:2454] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 0.
I20240503 02:13:20.050287 31390 leader_election.cc:302] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, fbb7d16e67d94775b2881712419c0577; no voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3
I20240503 02:13:20.050572 32271 raft_consensus.cc:2741] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 02:13:20.052378 31051 catalog_manager.cc:5484] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:20.401883 32303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:20.402187 32303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:20.409312 32303 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:13:20.450616 32283 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.458097 32283 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.552197 32253 consensus_queue.cc:1035] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.554495 32253 consensus_queue.cc:1035] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.573632 32328 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.578526 31584 raft_consensus.cc:3037] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.582652 31584 raft_consensus.cc:1261] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:20.583372 32328 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.633926 32338 mvcc.cc:205] Tried to move back new op lower bound from 7023421032762245120 to 7023421030606565376. Current Snapshot: MvccSnapshot[applied={T|T < 7023421032762245120}]
I20240503 02:13:20.653930 32343 mvcc.cc:205] Tried to move back new op lower bound from 7023421032762245120 to 7023421030606565376. Current Snapshot: MvccSnapshot[applied={T|T < 7023421032762245120}]
INSERT report
rows total: 2000
time total: 283.78 ms
time per row: 0.14189 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:13:21.738708 31043 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39754:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:21.746994 31304 tablet_service.cc:1467] Processing CreateTablet for tablet 8a5406bce7014bc2a7a1d727cb4aa191 (DEFAULT_TABLE table=rebalance_test_table_2 [id=88d4dd05cfa84299b1e62ec2a8b6a517]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.747141 31564 tablet_service.cc:1467] Processing CreateTablet for tablet 8a5406bce7014bc2a7a1d727cb4aa191 (DEFAULT_TABLE table=rebalance_test_table_2 [id=88d4dd05cfa84299b1e62ec2a8b6a517]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.747479 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a5406bce7014bc2a7a1d727cb4aa191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.747536 31564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a5406bce7014bc2a7a1d727cb4aa191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.748865 31172 tablet_service.cc:1467] Processing CreateTablet for tablet 8a5406bce7014bc2a7a1d727cb4aa191 (DEFAULT_TABLE table=rebalance_test_table_2 [id=88d4dd05cfa84299b1e62ec2a8b6a517]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.749181 31172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a5406bce7014bc2a7a1d727cb4aa191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.750512 31173 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8d17ee831446ae8d4a0417ede894b8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=88d4dd05cfa84299b1e62ec2a8b6a517]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.750882 31173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8d17ee831446ae8d4a0417ede894b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.754192 31302 tablet_service.cc:1467] Processing CreateTablet for tablet b76fa26b48e942ed840b5ee55ce40387 (DEFAULT_TABLE table=rebalance_test_table_2 [id=88d4dd05cfa84299b1e62ec2a8b6a517]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.754551 31302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76fa26b48e942ed840b5ee55ce40387. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.756379 31432 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8d17ee831446ae8d4a0417ede894b8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=88d4dd05cfa84299b1e62ec2a8b6a517]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.756906 31432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8d17ee831446ae8d4a0417ede894b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.757828 32374 tablet_bootstrap.cc:492] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:21.760226 31171 tablet_service.cc:1467] Processing CreateTablet for tablet b76fa26b48e942ed840b5ee55ce40387 (DEFAULT_TABLE table=rebalance_test_table_2 [id=88d4dd05cfa84299b1e62ec2a8b6a517]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.760704 32374 tablet_bootstrap.cc:654] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.760730 31171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76fa26b48e942ed840b5ee55ce40387. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.764364 32375 tablet_bootstrap.cc:492] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:21.764744 31564 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8d17ee831446ae8d4a0417ede894b8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=88d4dd05cfa84299b1e62ec2a8b6a517]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.765352 31564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8d17ee831446ae8d4a0417ede894b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.766626 32375 tablet_bootstrap.cc:654] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.764827 31563 tablet_service.cc:1467] Processing CreateTablet for tablet b76fa26b48e942ed840b5ee55ce40387 (DEFAULT_TABLE table=rebalance_test_table_2 [id=88d4dd05cfa84299b1e62ec2a8b6a517]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.772146 31563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76fa26b48e942ed840b5ee55ce40387. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.774484 32374 tablet_bootstrap.cc:492] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:21.774629 32374 ts_tablet_manager.cc:1367] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20240503 02:13:21.774900 32375 tablet_bootstrap.cc:492] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:21.775038 32375 ts_tablet_manager.cc:1367] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240503 02:13:21.775281 32374 raft_consensus.cc:348] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:21.775518 32374 raft_consensus.cc:374] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.775619 32374 raft_consensus.cc:729] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.775861 32374 consensus_queue.cc:260] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:21.775858 32375 raft_consensus.cc:348] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:21.776064 32375 raft_consensus.cc:374] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.776151 32375 raft_consensus.cc:729] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.776355 32374 ts_tablet_manager.cc:1398] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:21.776372 32375 consensus_queue.cc:260] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:21.777036 32375 ts_tablet_manager.cc:1398] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:21.777920 32375 tablet_bootstrap.cc:492] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:21.780328 32375 tablet_bootstrap.cc:654] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.789634 32374 tablet_bootstrap.cc:492] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:21.790820 32374 tablet_bootstrap.cc:654] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.792436 32381 tablet_bootstrap.cc:492] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:21.793095 32376 tablet_bootstrap.cc:492] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:21.794474 32381 tablet_bootstrap.cc:654] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.795190 32376 tablet_bootstrap.cc:654] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.796417 32375 tablet_bootstrap.cc:492] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:21.796561 32374 tablet_bootstrap.cc:492] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:21.796636 32375 ts_tablet_manager.cc:1367] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.005s
I20240503 02:13:21.796749 32374 ts_tablet_manager.cc:1367] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240503 02:13:21.796775 32381 tablet_bootstrap.cc:492] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:21.796891 32381 ts_tablet_manager.cc:1367] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:13:21.797482 32374 raft_consensus.cc:348] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:21.797590 32381 raft_consensus.cc:348] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:21.797572 32375 raft_consensus.cc:348] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:21.797791 32381 raft_consensus.cc:374] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.797827 32375 raft_consensus.cc:374] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.797734 32374 raft_consensus.cc:374] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.797869 32381 raft_consensus.cc:729] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.797920 32374 raft_consensus.cc:729] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.798007 32375 raft_consensus.cc:729] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.798229 32374 consensus_queue.cc:260] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:21.798390 32375 consensus_queue.cc:260] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:21.799353 32375 ts_tablet_manager.cc:1398] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:21.799619 32381 consensus_queue.cc:260] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:21.799770 32375 tablet_bootstrap.cc:492] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:21.801142 32381 ts_tablet_manager.cc:1398] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:13:21.802376 32375 tablet_bootstrap.cc:654] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.802546 32376 tablet_bootstrap.cc:492] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:21.802731 32376 ts_tablet_manager.cc:1367] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20240503 02:13:21.803615 32376 raft_consensus.cc:348] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:21.803869 32376 raft_consensus.cc:374] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.804015 32376 raft_consensus.cc:729] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.804267 32376 consensus_queue.cc:260] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:21.804963 32374 ts_tablet_manager.cc:1398] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:13:21.806010 32376 ts_tablet_manager.cc:1398] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:21.806305 32376 tablet_bootstrap.cc:492] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:21.808416 32376 tablet_bootstrap.cc:654] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.808823 32375 tablet_bootstrap.cc:492] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:21.808943 32375 ts_tablet_manager.cc:1367] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20240503 02:13:21.809590 32375 raft_consensus.cc:348] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:21.809742 32375 raft_consensus.cc:374] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.809849 32375 raft_consensus.cc:729] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.810076 32376 tablet_bootstrap.cc:492] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:21.810038 32375 consensus_queue.cc:260] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:21.810261 32376 ts_tablet_manager.cc:1367] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:13:21.810492 32375 ts_tablet_manager.cc:1398] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 02:13:21.810964 32376 raft_consensus.cc:348] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:21.811167 32376 raft_consensus.cc:374] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.811244 32376 raft_consensus.cc:729] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.811434 32376 consensus_queue.cc:260] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:21.811811 32376 ts_tablet_manager.cc:1398] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:13:21.812108 32376 tablet_bootstrap.cc:492] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:21.813421 32376 tablet_bootstrap.cc:654] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.815253 32376 tablet_bootstrap.cc:492] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:21.815393 32376 ts_tablet_manager.cc:1367] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:21.815912 32376 raft_consensus.cc:348] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:21.816041 32376 raft_consensus.cc:374] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.816104 32376 raft_consensus.cc:729] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.816272 32376 consensus_queue.cc:260] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:21.816655 32376 ts_tablet_manager.cc:1398] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:13:21.921082 32328 raft_consensus.cc:482] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:21.921252 32328 raft_consensus.cc:504] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:21.921824 32328 leader_election.cc:288] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:21.922240 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76fa26b48e942ed840b5ee55ce40387" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:21.922288 31584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76fa26b48e942ed840b5ee55ce40387" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3" is_pre_election: true
I20240503 02:13:21.922464 31192 raft_consensus.cc:2454] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:21.922487 31584 raft_consensus.cc:2454] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:21.922870 31259 leader_election.cc:302] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:21.923132 32328 raft_consensus.cc:2781] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:21.923255 32328 raft_consensus.cc:482] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:21.923324 32328 raft_consensus.cc:3037] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.925212 32328 raft_consensus.cc:504] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:21.925750 32328 leader_election.cc:288] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Requested vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:21.926137 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76fa26b48e942ed840b5ee55ce40387" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:21.926224 31584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76fa26b48e942ed840b5ee55ce40387" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:21.926326 31192 raft_consensus.cc:3037] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.926422 31584 raft_consensus.cc:3037] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.928179 31192 raft_consensus.cc:2454] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:21.928344 31584 raft_consensus.cc:2454] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:21.928620 31259 leader_election.cc:302] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:21.929000 32328 raft_consensus.cc:2781] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:21.929353 32328 raft_consensus.cc:686] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Leader. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:21.929611 32328 consensus_queue.cc:237] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:21.932163 31051 catalog_manager.cc:5484] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:21.983593 32350 raft_consensus.cc:482] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:21.983824 32350 raft_consensus.cc:504] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:21.984573 32350 leader_election.cc:288] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:21.985044 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5406bce7014bc2a7a1d727cb4aa191" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:21.985042 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5406bce7014bc2a7a1d727cb4aa191" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" is_pre_election: true
I20240503 02:13:21.985283 31323 raft_consensus.cc:2454] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 0.
I20240503 02:13:21.985280 31192 raft_consensus.cc:2454] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 0.
I20240503 02:13:21.985706 31519 leader_election.cc:302] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, b92ff49cec804b1f801c11b64e3259b3; no voters:
I20240503 02:13:21.985950 32350 raft_consensus.cc:2781] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:21.986083 32350 raft_consensus.cc:482] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:21.986168 32350 raft_consensus.cc:3037] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.988588 32350 raft_consensus.cc:504] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:21.989073 32350 leader_election.cc:288] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Requested vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:21.989421 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5406bce7014bc2a7a1d727cb4aa191" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
I20240503 02:13:21.989568 31323 raft_consensus.cc:3037] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.989486 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5406bce7014bc2a7a1d727cb4aa191" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:21.989691 31192 raft_consensus.cc:3037] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.991317 31323 raft_consensus.cc:2454] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 1.
I20240503 02:13:21.991631 31192 raft_consensus.cc:2454] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 1.
I20240503 02:13:21.991633 31518 leader_election.cc:302] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:21.992013 32350 raft_consensus.cc:2781] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:21.992153 32350 raft_consensus.cc:686] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Becoming Leader. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Running, Role: LEADER
I20240503 02:13:21.992406 32350 consensus_queue.cc:237] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:21.995872 31051 catalog_manager.cc:5484] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196). New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:21.997253 32350 raft_consensus.cc:482] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:21.997819 32350 raft_consensus.cc:504] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:21.998898 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8d17ee831446ae8d4a0417ede894b8" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:21.998926 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8d17ee831446ae8d4a0417ede894b8" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:21.999192 31454 raft_consensus.cc:2454] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 0.
I20240503 02:13:21.999177 31192 raft_consensus.cc:2454] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 0.
I20240503 02:13:21.999572 31519 leader_election.cc:302] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, b92ff49cec804b1f801c11b64e3259b3; no voters:
I20240503 02:13:21.999992 32369 raft_consensus.cc:2781] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:22.000087 32369 raft_consensus.cc:482] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:22.000211 32369 raft_consensus.cc:3037] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.002640 32369 raft_consensus.cc:504] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:22.003160 32369 leader_election.cc:288] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Requested vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:22.003302 32350 leader_election.cc:288] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:22.003576 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8d17ee831446ae8d4a0417ede894b8" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:22.003636 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8d17ee831446ae8d4a0417ede894b8" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:22.003743 31454 raft_consensus.cc:3037] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.003840 31192 raft_consensus.cc:3037] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.005514 31454 raft_consensus.cc:2454] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 1.
I20240503 02:13:22.005838 31517 leader_election.cc:302] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:22.006078 32350 raft_consensus.cc:2781] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:22.006186 32350 raft_consensus.cc:686] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Becoming Leader. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Running, Role: LEADER
I20240503 02:13:22.006410 32350 consensus_queue.cc:237] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:22.006939 31192 raft_consensus.cc:2454] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 1.
I20240503 02:13:22.008653 31051 catalog_manager.cc:5484] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196). New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:22.320226 32328 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.331151 32388 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:22.386385 32392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:22.386708 32392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:22.394017 32392 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:22.423533 32350 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 02:13:22.429476 32350 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.449939 32369 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.454792 32369 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:22.651005 31519 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:44019, user_credentials={real_user=slave}} blocked reactor thread for 61551.6us
INSERT report
rows total: 2000
time total: 265.129 ms
time per row: 0.132565 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:13:23.723910 31051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39754:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:23.733415 31172 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3d978cd00449b8b3edd4c43b657960 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e0a6e74497494a20be21d627357e4eb0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.733916 31173 tablet_service.cc:1467] Processing CreateTablet for tablet 091475e28bb344e78b1e4e48fa548ef8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e0a6e74497494a20be21d627357e4eb0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.734313 31563 tablet_service.cc:1467] Processing CreateTablet for tablet 091475e28bb344e78b1e4e48fa548ef8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e0a6e74497494a20be21d627357e4eb0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.734724 31563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 091475e28bb344e78b1e4e48fa548ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.734634 31304 tablet_service.cc:1467] Processing CreateTablet for tablet 17be048393974fdf9811c5b8f8a97579 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e0a6e74497494a20be21d627357e4eb0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.735028 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17be048393974fdf9811c5b8f8a97579. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.734997 31564 tablet_service.cc:1467] Processing CreateTablet for tablet 17be048393974fdf9811c5b8f8a97579 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e0a6e74497494a20be21d627357e4eb0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.735149 31432 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3d978cd00449b8b3edd4c43b657960 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e0a6e74497494a20be21d627357e4eb0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.735338 31564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17be048393974fdf9811c5b8f8a97579. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.735561 31432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3d978cd00449b8b3edd4c43b657960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.740154 31433 tablet_service.cc:1467] Processing CreateTablet for tablet 091475e28bb344e78b1e4e48fa548ef8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e0a6e74497494a20be21d627357e4eb0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.734634 31302 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3d978cd00449b8b3edd4c43b657960 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e0a6e74497494a20be21d627357e4eb0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.740504 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 091475e28bb344e78b1e4e48fa548ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.740696 31302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3d978cd00449b8b3edd4c43b657960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.744279 31434 tablet_service.cc:1467] Processing CreateTablet for tablet 17be048393974fdf9811c5b8f8a97579 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e0a6e74497494a20be21d627357e4eb0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.744737 31434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17be048393974fdf9811c5b8f8a97579. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.733938 31172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3d978cd00449b8b3edd4c43b657960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.749732 32461 tablet_bootstrap.cc:492] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:23.750006 32462 tablet_bootstrap.cc:492] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:23.751114 32462 tablet_bootstrap.cc:654] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.751142 32461 tablet_bootstrap.cc:654] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.752152 31173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 091475e28bb344e78b1e4e48fa548ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.752776 32462 tablet_bootstrap.cc:492] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:23.752908 32462 ts_tablet_manager.cc:1367] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:23.753472 32462 raft_consensus.cc:348] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.753644 32462 raft_consensus.cc:374] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.753746 32462 raft_consensus.cc:729] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.753955 32462 consensus_queue.cc:260] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.754426 32462 ts_tablet_manager.cc:1398] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:13:23.756435 32461 tablet_bootstrap.cc:492] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:23.756574 32461 ts_tablet_manager.cc:1367] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:13:23.757292 32461 raft_consensus.cc:348] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:23.757483 32461 raft_consensus.cc:374] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.757560 32461 raft_consensus.cc:729] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.757759 32461 consensus_queue.cc:260] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:23.758312 32461 ts_tablet_manager.cc:1398] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:23.758566 32461 tablet_bootstrap.cc:492] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:23.760272 32461 tablet_bootstrap.cc:654] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.762303 32461 tablet_bootstrap.cc:492] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:23.762465 32461 ts_tablet_manager.cc:1367] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:13:23.763151 32461 raft_consensus.cc:348] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.763357 32461 raft_consensus.cc:374] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.763442 32461 raft_consensus.cc:729] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.763684 32461 consensus_queue.cc:260] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.764235 32461 ts_tablet_manager.cc:1398] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:23.767933 32466 tablet_bootstrap.cc:492] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:23.769657 32466 tablet_bootstrap.cc:654] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.774547 32462 tablet_bootstrap.cc:492] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:23.776190 32462 tablet_bootstrap.cc:654] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.777032 32460 tablet_bootstrap.cc:492] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:23.777545 32466 tablet_bootstrap.cc:492] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:23.777722 32466 ts_tablet_manager.cc:1367] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 02:13:23.778147 32462 tablet_bootstrap.cc:492] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:23.778309 32462 ts_tablet_manager.cc:1367] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:13:23.778331 32466 raft_consensus.cc:348] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.778506 32466 raft_consensus.cc:374] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.778568 32460 tablet_bootstrap.cc:654] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.778616 32466 raft_consensus.cc:729] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.778856 32466 consensus_queue.cc:260] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.779119 32462 raft_consensus.cc:348] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.779335 32462 raft_consensus.cc:374] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.779438 32462 raft_consensus.cc:729] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.779878 32462 consensus_queue.cc:260] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.780622 32462 ts_tablet_manager.cc:1398] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 02:13:23.781435 32462 tablet_bootstrap.cc:492] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:23.782105 32466 ts_tablet_manager.cc:1398] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:23.782626 32466 tablet_bootstrap.cc:492] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:23.782954 32462 tablet_bootstrap.cc:654] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.784109 32466 tablet_bootstrap.cc:654] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.785846 32466 tablet_bootstrap.cc:492] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:23.785989 32466 ts_tablet_manager.cc:1367] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:23.786696 32466 raft_consensus.cc:348] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.786814 32462 tablet_bootstrap.cc:492] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:23.786866 32466 raft_consensus.cc:374] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.786958 32466 raft_consensus.cc:729] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.786981 32462 ts_tablet_manager.cc:1367] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 02:13:23.787195 32466 consensus_queue.cc:260] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.787744 32466 ts_tablet_manager.cc:1398] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:23.787927 32462 raft_consensus.cc:348] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:23.788178 32462 raft_consensus.cc:374] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.789702 32462 raft_consensus.cc:729] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.789940 32462 consensus_queue.cc:260] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:23.790131 32460 tablet_bootstrap.cc:492] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:23.790282 32460 ts_tablet_manager.cc:1367] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20240503 02:13:23.790480 32462 ts_tablet_manager.cc:1398] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:23.790937 32460 raft_consensus.cc:348] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:23.791133 32460 raft_consensus.cc:374] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.791240 32460 raft_consensus.cc:729] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.791484 32460 consensus_queue.cc:260] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:23.792551 32460 ts_tablet_manager.cc:1398] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:23.792816 32460 tablet_bootstrap.cc:492] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:23.793982 32460 tablet_bootstrap.cc:654] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.795953 32460 tablet_bootstrap.cc:492] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:23.796090 32460 ts_tablet_manager.cc:1367] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:23.796566 32460 raft_consensus.cc:348] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.796725 32460 raft_consensus.cc:374] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.796794 32460 raft_consensus.cc:729] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.796986 32460 consensus_queue.cc:260] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.797431 32460 ts_tablet_manager.cc:1398] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:13:23.880677 32271 raft_consensus.cc:482] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:23.880841 32271 raft_consensus.cc:504] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.881449 32271 leader_election.cc:288] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:23.881804 32369 raft_consensus.cc:482] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:23.881804 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3d978cd00449b8b3edd4c43b657960" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" is_pre_election: true
I20240503 02:13:23.882023 31323 raft_consensus.cc:2454] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 0.
I20240503 02:13:23.881978 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3d978cd00449b8b3edd4c43b657960" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:23.882009 32369 raft_consensus.cc:504] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.882220 31193 raft_consensus.cc:2454] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 0.
I20240503 02:13:23.882429 31388 leader_election.cc:302] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:23.882681 32369 leader_election.cc:288] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:23.882772 32271 raft_consensus.cc:2781] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:23.882895 32271 raft_consensus.cc:482] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:23.882988 32271 raft_consensus.cc:3037] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.883201 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091475e28bb344e78b1e4e48fa548ef8" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:23.883376 31193 raft_consensus.cc:2454] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 0.
I20240503 02:13:23.883324 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091475e28bb344e78b1e4e48fa548ef8" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:23.883508 31454 raft_consensus.cc:2454] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 0.
I20240503 02:13:23.883700 31519 leader_election.cc:302] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, b92ff49cec804b1f801c11b64e3259b3; no voters:
I20240503 02:13:23.884109 32369 raft_consensus.cc:2781] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:23.884222 32369 raft_consensus.cc:482] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:23.884330 32369 raft_consensus.cc:3037] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.885962 32271 raft_consensus.cc:504] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.886560 32271 leader_election.cc:288] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 election: Requested vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:23.886879 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3d978cd00449b8b3edd4c43b657960" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
I20240503 02:13:23.887022 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3d978cd00449b8b3edd4c43b657960" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:23.887073 31323 raft_consensus.cc:3037] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.886966 32369 raft_consensus.cc:504] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.887215 31193 raft_consensus.cc:3037] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.887684 32369 leader_election.cc:288] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Requested vote from peers fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:23.889061 31323 raft_consensus.cc:2454] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 1.
I20240503 02:13:23.889185 31193 raft_consensus.cc:2454] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 1.
I20240503 02:13:23.889473 31388 leader_election.cc:302] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:23.889570 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091475e28bb344e78b1e4e48fa548ef8" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:23.889732 31193 raft_consensus.cc:3037] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.889777 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091475e28bb344e78b1e4e48fa548ef8" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:23.889981 31454 raft_consensus.cc:3037] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.890055 32271 raft_consensus.cc:2781] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:23.890551 32271 raft_consensus.cc:686] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Becoming Leader. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Running, Role: LEADER
I20240503 02:13:23.890904 32271 consensus_queue.cc:237] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.892338 31193 raft_consensus.cc:2454] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 1.
I20240503 02:13:23.892577 31454 raft_consensus.cc:2454] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 1.
I20240503 02:13:23.892700 31519 leader_election.cc:302] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, b92ff49cec804b1f801c11b64e3259b3; no voters:
I20240503 02:13:23.892978 32369 raft_consensus.cc:2781] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:23.893374 32369 raft_consensus.cc:686] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Becoming Leader. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Running, Role: LEADER
I20240503 02:13:23.893630 32369 consensus_queue.cc:237] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:23.894702 31051 catalog_manager.cc:5484] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbb7d16e67d94775b2881712419c0577 (127.23.104.195). New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:23.896325 31043 catalog_manager.cc:5484] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196). New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:24.202157 32369 raft_consensus.cc:482] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:24.202360 32369 raft_consensus.cc:504] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:24.203042 32369 leader_election.cc:288] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:24.203553 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17be048393974fdf9811c5b8f8a97579" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:24.203567 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17be048393974fdf9811c5b8f8a97579" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" is_pre_election: true
I20240503 02:13:24.203827 31454 raft_consensus.cc:2454] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 0.
I20240503 02:13:24.203882 31323 raft_consensus.cc:2454] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 0.
I20240503 02:13:24.204239 31517 leader_election.cc:302] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:24.204476 32369 raft_consensus.cc:2781] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:24.204586 32369 raft_consensus.cc:482] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:24.204651 32369 raft_consensus.cc:3037] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:24.206475 32369 raft_consensus.cc:504] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:24.207015 32369 leader_election.cc:288] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Requested vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:24.207393 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17be048393974fdf9811c5b8f8a97579" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
I20240503 02:13:24.207453 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17be048393974fdf9811c5b8f8a97579" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:24.207610 31323 raft_consensus.cc:3037] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:24.207641 31454 raft_consensus.cc:3037] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:24.210444 31454 raft_consensus.cc:2454] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 1.
I20240503 02:13:24.210458 31323 raft_consensus.cc:2454] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 1.
I20240503 02:13:24.210803 31518 leader_election.cc:302] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:24.211081 32369 raft_consensus.cc:2781] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:24.211202 32369 raft_consensus.cc:686] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Becoming Leader. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Running, Role: LEADER
I20240503 02:13:24.211510 32369 consensus_queue.cc:237] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:24.214076 31043 catalog_manager.cc:5484] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196). New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:24.272960 32369 consensus_queue.cc:1035] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:24.277559 32476 consensus_queue.cc:1035] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:24.335155 32271 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:24.341509 32271 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:24.641093 32476 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:24.646139 32369 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:24.657245 32484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:24.657496 32484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:24.664376 32484 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 211.586 ms
time per row: 0.105793 ms
I20240503 02:13:25.027920 31043 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39754:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:25.036861 31304 tablet_service.cc:1467] Processing CreateTablet for tablet 8c681bdcf3074c54b1d533cf1f7d2b75 (DEFAULT_TABLE table=rebalance_test_table_4 [id=38897cba33374498b288d375d22641bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:25.037168 31172 tablet_service.cc:1467] Processing CreateTablet for tablet 8c681bdcf3074c54b1d533cf1f7d2b75 (DEFAULT_TABLE table=rebalance_test_table_4 [id=38897cba33374498b288d375d22641bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:25.037339 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c681bdcf3074c54b1d533cf1f7d2b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:25.037550 31172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c681bdcf3074c54b1d533cf1f7d2b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:25.038403 31563 tablet_service.cc:1467] Processing CreateTablet for tablet 1c425b03a1c64d7ba7a14901d63d7037 (DEFAULT_TABLE table=rebalance_test_table_4 [id=38897cba33374498b288d375d22641bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:25.038862 31563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c425b03a1c64d7ba7a14901d63d7037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:25.038945 31434 tablet_service.cc:1467] Processing CreateTablet for tablet 8c681bdcf3074c54b1d533cf1f7d2b75 (DEFAULT_TABLE table=rebalance_test_table_4 [id=38897cba33374498b288d375d22641bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:25.039350 31434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c681bdcf3074c54b1d533cf1f7d2b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:25.040833 31564 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc576d8e72647c08736ae9508f6d27b (DEFAULT_TABLE table=rebalance_test_table_4 [id=38897cba33374498b288d375d22641bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:25.041205 31564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc576d8e72647c08736ae9508f6d27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:25.044142 31302 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc576d8e72647c08736ae9508f6d27b (DEFAULT_TABLE table=rebalance_test_table_4 [id=38897cba33374498b288d375d22641bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:25.044495 31302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc576d8e72647c08736ae9508f6d27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:25.045127 31173 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc576d8e72647c08736ae9508f6d27b (DEFAULT_TABLE table=rebalance_test_table_4 [id=38897cba33374498b288d375d22641bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:25.045518 31173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc576d8e72647c08736ae9508f6d27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:25.048223 31171 tablet_service.cc:1467] Processing CreateTablet for tablet 1c425b03a1c64d7ba7a14901d63d7037 (DEFAULT_TABLE table=rebalance_test_table_4 [id=38897cba33374498b288d375d22641bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:25.048574 31171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c425b03a1c64d7ba7a14901d63d7037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:25.051401 32547 tablet_bootstrap.cc:492] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:25.052153 31432 tablet_service.cc:1467] Processing CreateTablet for tablet 1c425b03a1c64d7ba7a14901d63d7037 (DEFAULT_TABLE table=rebalance_test_table_4 [id=38897cba33374498b288d375d22641bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:25.052520 31432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c425b03a1c64d7ba7a14901d63d7037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:25.053056 32547 tablet_bootstrap.cc:654] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:25.055960 32545 tablet_bootstrap.cc:492] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:25.057664 32545 tablet_bootstrap.cc:654] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:25.059324 32547 tablet_bootstrap.cc:492] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:25.059501 32547 ts_tablet_manager.cc:1367] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 02:13:25.060509 32547 raft_consensus.cc:348] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:25.060956 32547 raft_consensus.cc:374] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:25.061199 32549 tablet_bootstrap.cc:492] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:25.061095 32547 raft_consensus.cc:729] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:25.061558 32547 consensus_queue.cc:260] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:25.062645 32547 ts_tablet_manager.cc:1398] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 02:13:25.063089 32546 tablet_bootstrap.cc:492] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:25.064280 32547 tablet_bootstrap.cc:492] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:25.064687 32546 tablet_bootstrap.cc:654] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:25.065881 32545 tablet_bootstrap.cc:492] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:25.065925 32547 tablet_bootstrap.cc:654] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:25.066098 32545 ts_tablet_manager.cc:1367] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240503 02:13:25.067044 32545 raft_consensus.cc:348] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:25.067370 32545 raft_consensus.cc:374] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:25.067530 32545 raft_consensus.cc:729] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:25.067967 32545 consensus_queue.cc:260] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:25.068459 32549 tablet_bootstrap.cc:654] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:25.071367 32545 ts_tablet_manager.cc:1398] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:25.073295 32546 tablet_bootstrap.cc:492] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:25.073516 32546 ts_tablet_manager.cc:1367] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240503 02:13:25.074457 32545 tablet_bootstrap.cc:492] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:25.074532 32546 raft_consensus.cc:348] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:25.074852 32546 raft_consensus.cc:374] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:25.075001 32546 raft_consensus.cc:729] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:25.075362 32546 consensus_queue.cc:260] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:25.076649 32545 tablet_bootstrap.cc:654] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:25.076844 32549 tablet_bootstrap.cc:492] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:25.077071 32549 ts_tablet_manager.cc:1367] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20240503 02:13:25.078145 32549 raft_consensus.cc:348] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:25.078455 32549 raft_consensus.cc:374] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:25.078630 32549 raft_consensus.cc:729] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:25.079075 32549 consensus_queue.cc:260] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:25.080107 32546 ts_tablet_manager.cc:1398] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20240503 02:13:25.080610 32549 ts_tablet_manager.cc:1398] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:25.080626 32547 tablet_bootstrap.cc:492] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:25.080780 32547 ts_tablet_manager.cc:1367] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.001s
I20240503 02:13:25.080935 32549 tablet_bootstrap.cc:492] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:25.080962 32546 tablet_bootstrap.cc:492] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:25.081475 32547 raft_consensus.cc:348] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:25.081709 32547 raft_consensus.cc:374] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:25.081789 32547 raft_consensus.cc:729] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:25.082012 32547 consensus_queue.cc:260] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:25.082544 32546 tablet_bootstrap.cc:654] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:25.082556 32547 ts_tablet_manager.cc:1398] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:25.082564 32549 tablet_bootstrap.cc:654] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:25.083990 32545 tablet_bootstrap.cc:492] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:25.084142 32545 ts_tablet_manager.cc:1367] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240503 02:13:25.084676 32546 tablet_bootstrap.cc:492] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:25.084826 32546 ts_tablet_manager.cc:1367] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:13:25.084883 32545 raft_consensus.cc:348] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:25.085100 32545 raft_consensus.cc:374] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:25.085183 32545 raft_consensus.cc:729] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:25.085438 32545 consensus_queue.cc:260] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:25.085539 32546 raft_consensus.cc:348] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:25.085742 32546 raft_consensus.cc:374] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:25.085830 32546 raft_consensus.cc:729] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:25.086063 32546 consensus_queue.cc:260] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:25.086565 32546 ts_tablet_manager.cc:1398] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:25.086690 32545 ts_tablet_manager.cc:1398] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:25.086848 32546 tablet_bootstrap.cc:492] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:25.090029 32546 tablet_bootstrap.cc:654] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:25.092579 32549 tablet_bootstrap.cc:492] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:25.092633 32546 tablet_bootstrap.cc:492] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:25.092744 32546 ts_tablet_manager.cc:1367] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 02:13:25.092758 32549 ts_tablet_manager.cc:1367] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240503 02:13:25.093415 32546 raft_consensus.cc:348] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:25.093406 32549 raft_consensus.cc:348] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:25.093617 32546 raft_consensus.cc:374] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:25.093644 32549 raft_consensus.cc:374] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:25.093729 32546 raft_consensus.cc:729] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:25.093725 32549 raft_consensus.cc:729] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:25.093950 32546 consensus_queue.cc:260] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:25.093959 32549 consensus_queue.cc:260] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:25.094424 32546 ts_tablet_manager.cc:1398] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:25.094446 32549 ts_tablet_manager.cc:1398] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:25.206193 32397 raft_consensus.cc:482] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:25.206398 32397 raft_consensus.cc:504] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:25.207154 32397 leader_election.cc:288] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:25.207612 31584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc576d8e72647c08736ae9508f6d27b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3" is_pre_election: true
I20240503 02:13:25.207674 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc576d8e72647c08736ae9508f6d27b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:25.207870 31584 raft_consensus.cc:2454] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:25.207979 31192 raft_consensus.cc:2454] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:25.208338 31260 leader_election.cc:302] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:25.208576 32397 raft_consensus.cc:2781] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:25.208678 32397 raft_consensus.cc:482] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:25.208774 32397 raft_consensus.cc:3037] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:25.210866 32397 raft_consensus.cc:504] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:25.211366 32397 leader_election.cc:288] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Requested vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:25.211778 31584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc576d8e72647c08736ae9508f6d27b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:25.211874 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc576d8e72647c08736ae9508f6d27b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:25.212050 31584 raft_consensus.cc:3037] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:25.212049 31192 raft_consensus.cc:3037] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:25.214797 31192 raft_consensus.cc:2454] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:25.214785 31584 raft_consensus.cc:2454] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:25.215152 31260 leader_election.cc:302] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:25.215401 32397 raft_consensus.cc:2781] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:25.215832 32397 raft_consensus.cc:686] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Leader. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:25.216305 32397 consensus_queue.cc:237] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:25.218833 31044 catalog_manager.cc:5484] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:25.275740 32271 raft_consensus.cc:482] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:25.275957 32271 raft_consensus.cc:504] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:25.276511 32271 leader_election.cc:288] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:25.276993 31584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c425b03a1c64d7ba7a14901d63d7037" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3" is_pre_election: true
I20240503 02:13:25.277020 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c425b03a1c64d7ba7a14901d63d7037" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:25.277238 31584 raft_consensus.cc:2454] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 0.
I20240503 02:13:25.277271 31192 raft_consensus.cc:2454] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 0.
I20240503 02:13:25.277690 31389 leader_election.cc:302] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:25.277963 32271 raft_consensus.cc:2781] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:25.278045 32271 raft_consensus.cc:482] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:25.278128 32271 raft_consensus.cc:3037] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:25.279940 32271 raft_consensus.cc:504] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:25.280473 32271 leader_election.cc:288] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 election: Requested vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:25.280779 31584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c425b03a1c64d7ba7a14901d63d7037" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:25.280962 31584 raft_consensus.cc:3037] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:25.281001 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c425b03a1c64d7ba7a14901d63d7037" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:25.281205 31192 raft_consensus.cc:3037] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:25.283828 31584 raft_consensus.cc:2454] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 1.
I20240503 02:13:25.284005 31192 raft_consensus.cc:2454] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 1.
I20240503 02:13:25.284260 31390 leader_election.cc:302] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:25.284545 32271 raft_consensus.cc:2781] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:25.284698 32271 raft_consensus.cc:686] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Becoming Leader. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Running, Role: LEADER
I20240503 02:13:25.284957 32271 consensus_queue.cc:237] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:25.287420 31044 catalog_manager.cc:5484] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbb7d16e67d94775b2881712419c0577 (127.23.104.195). New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:25.407850 32397 raft_consensus.cc:482] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:25.408104 32397 raft_consensus.cc:504] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:25.409004 32397 leader_election.cc:288] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:25.409178 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c681bdcf3074c54b1d533cf1f7d2b75" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:25.409266 31453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c681bdcf3074c54b1d533cf1f7d2b75" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:25.409376 31192 raft_consensus.cc:2454] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:25.409523 31453 raft_consensus.cc:2454] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:25.409722 31259 leader_election.cc:302] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:25.410058 32397 raft_consensus.cc:2781] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:25.410146 32397 raft_consensus.cc:482] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:25.410243 32397 raft_consensus.cc:3037] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:25.411993 32397 raft_consensus.cc:504] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:25.412508 32397 leader_election.cc:288] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Requested vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:25.412808 31192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c681bdcf3074c54b1d533cf1f7d2b75" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:25.412899 31453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c681bdcf3074c54b1d533cf1f7d2b75" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:25.412998 31192 raft_consensus.cc:3037] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:25.413067 31453 raft_consensus.cc:3037] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:25.415678 31192 raft_consensus.cc:2454] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:25.415710 31453 raft_consensus.cc:2454] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:25.416056 31257 leader_election.cc:302] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:25.416311 32397 raft_consensus.cc:2781] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:25.416492 32397 raft_consensus.cc:686] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Leader. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:25.416749 32397 consensus_queue.cc:237] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:25.419454 31044 catalog_manager.cc:5484] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:25.671926 32397 consensus_queue.cc:1035] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:25.679003 32397 consensus_queue.cc:1035] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:25.685277 32271 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:25.690279 32271 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:25.829556 32563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:25.829874 32563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:25.837126 32563 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 02:13:25.955188 32397 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:25.983350 32397 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 236.715 ms
time per row: 0.118357 ms
I20240503 02:13:26.213467 31049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39754:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:26.221371 31434 tablet_service.cc:1467] Processing CreateTablet for tablet 9000ce76eb154434a88b52e357497767 (DEFAULT_TABLE table=rebalance_test_table_5 [id=f64903d295f2485983d7a34ee454f2b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:26.221828 31434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9000ce76eb154434a88b52e357497767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.222092 31304 tablet_service.cc:1467] Processing CreateTablet for tablet 9000ce76eb154434a88b52e357497767 (DEFAULT_TABLE table=rebalance_test_table_5 [id=f64903d295f2485983d7a34ee454f2b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:26.222506 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9000ce76eb154434a88b52e357497767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.223093 31563 tablet_service.cc:1467] Processing CreateTablet for tablet 7626c06dc67448f0b65f67b401d62c88 (DEFAULT_TABLE table=rebalance_test_table_5 [id=f64903d295f2485983d7a34ee454f2b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:26.223479 31563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7626c06dc67448f0b65f67b401d62c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.224145 31564 tablet_service.cc:1467] Processing CreateTablet for tablet 9000ce76eb154434a88b52e357497767 (DEFAULT_TABLE table=rebalance_test_table_5 [id=f64903d295f2485983d7a34ee454f2b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:26.224483 31564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9000ce76eb154434a88b52e357497767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.227265 31302 tablet_service.cc:1467] Processing CreateTablet for tablet 7626c06dc67448f0b65f67b401d62c88 (DEFAULT_TABLE table=rebalance_test_table_5 [id=f64903d295f2485983d7a34ee454f2b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:26.227639 31302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7626c06dc67448f0b65f67b401d62c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.228256 31172 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8e7774779e47ba9d426ac00ad8be62 (DEFAULT_TABLE table=rebalance_test_table_5 [id=f64903d295f2485983d7a34ee454f2b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:26.228652 31172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8e7774779e47ba9d426ac00ad8be62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.230940 32628 tablet_bootstrap.cc:492] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:26.232203 32628 tablet_bootstrap.cc:654] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:26.232532 31303 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8e7774779e47ba9d426ac00ad8be62 (DEFAULT_TABLE table=rebalance_test_table_5 [id=f64903d295f2485983d7a34ee454f2b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:26.232633 31432 tablet_service.cc:1467] Processing CreateTablet for tablet 7626c06dc67448f0b65f67b401d62c88 (DEFAULT_TABLE table=rebalance_test_table_5 [id=f64903d295f2485983d7a34ee454f2b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:26.232808 31433 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8e7774779e47ba9d426ac00ad8be62 (DEFAULT_TABLE table=rebalance_test_table_5 [id=f64903d295f2485983d7a34ee454f2b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:26.232920 31303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8e7774779e47ba9d426ac00ad8be62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.233031 31432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7626c06dc67448f0b65f67b401d62c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.233440 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8e7774779e47ba9d426ac00ad8be62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.246162 32631 tablet_bootstrap.cc:492] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:26.248169 32630 tablet_bootstrap.cc:492] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:26.248503 32631 tablet_bootstrap.cc:654] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:26.248575 32627 tablet_bootstrap.cc:492] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:26.249171 32628 tablet_bootstrap.cc:492] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:26.249424 32628 ts_tablet_manager.cc:1367] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20240503 02:13:26.250306 32630 tablet_bootstrap.cc:654] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:26.250250 32628 raft_consensus.cc:348] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:26.250481 32628 raft_consensus.cc:374] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:26.250593 32627 tablet_bootstrap.cc:654] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:26.250597 32628 raft_consensus.cc:729] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:26.250844 32628 consensus_queue.cc:260] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:26.252796 32628 ts_tablet_manager.cc:1398] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:13:26.253171 32628 tablet_bootstrap.cc:492] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:26.255220 32628 tablet_bootstrap.cc:654] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:26.259025 32630 tablet_bootstrap.cc:492] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:26.259259 32630 ts_tablet_manager.cc:1367] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240503 02:13:26.260401 32630 raft_consensus.cc:348] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:26.260711 32630 raft_consensus.cc:374] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:26.260855 32630 raft_consensus.cc:729] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:26.261251 32630 consensus_queue.cc:260] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:26.262892 32630 ts_tablet_manager.cc:1398] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:26.263267 32630 tablet_bootstrap.cc:492] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:26.263828 32627 tablet_bootstrap.cc:492] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:26.263959 32627 ts_tablet_manager.cc:1367] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.001s
I20240503 02:13:26.264840 32627 raft_consensus.cc:348] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:26.265024 32627 raft_consensus.cc:374] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:26.265107 32627 raft_consensus.cc:729] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:26.265313 32627 consensus_queue.cc:260] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:26.266404 32630 tablet_bootstrap.cc:654] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:26.266892 32628 tablet_bootstrap.cc:492] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:26.267115 32628 ts_tablet_manager.cc:1367] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240503 02:13:26.268038 32628 raft_consensus.cc:348] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:26.268344 32628 raft_consensus.cc:374] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:26.268455 32628 raft_consensus.cc:729] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:26.268785 32628 consensus_queue.cc:260] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:26.268988 32627 ts_tablet_manager.cc:1398] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:13:26.270807 32628 ts_tablet_manager.cc:1398] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:26.271075 32627 tablet_bootstrap.cc:492] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:26.274246 32631 tablet_bootstrap.cc:492] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:26.274392 32631 ts_tablet_manager.cc:1367] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.022s
I20240503 02:13:26.274947 32631 raft_consensus.cc:348] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:26.275096 32631 raft_consensus.cc:374] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:26.275178 32631 raft_consensus.cc:729] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:26.275358 32631 consensus_queue.cc:260] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:26.275904 32631 ts_tablet_manager.cc:1398] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:13:26.276162 32627 tablet_bootstrap.cc:654] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:26.276729 32630 tablet_bootstrap.cc:492] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:26.276909 32630 ts_tablet_manager.cc:1367] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.005s
I20240503 02:13:26.277714 32630 raft_consensus.cc:348] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:26.278023 32630 raft_consensus.cc:374] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:26.278160 32630 raft_consensus.cc:729] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:26.278450 32630 consensus_queue.cc:260] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:26.279006 32627 tablet_bootstrap.cc:492] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:26.279033 32630 ts_tablet_manager.cc:1398] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:26.279203 32627 ts_tablet_manager.cc:1367] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20240503 02:13:26.279528 32630 tablet_bootstrap.cc:492] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:26.279573 32474 raft_consensus.cc:482] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:26.279724 32474 raft_consensus.cc:504] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:26.280526 32474 leader_election.cc:288] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:26.280946 31324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9000ce76eb154434a88b52e357497767" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" is_pre_election: true
I20240503 02:13:26.281116 32630 tablet_bootstrap.cc:654] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:26.281180 31324 raft_consensus.cc:2454] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 0.
I20240503 02:13:26.281600 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9000ce76eb154434a88b52e357497767" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3" is_pre_election: true
I20240503 02:13:26.280001 32627 raft_consensus.cc:348] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:26.281805 32627 raft_consensus.cc:374] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:26.281807 31582 raft_consensus.cc:2454] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 0.
I20240503 02:13:26.281886 32627 raft_consensus.cc:729] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:26.282127 32627 consensus_queue.cc:260] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:26.282330 31390 leader_election.cc:302] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:26.283004 32474 raft_consensus.cc:2781] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:26.283154 32474 raft_consensus.cc:482] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:26.283255 32474 raft_consensus.cc:3037] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:26.285005 32627 ts_tablet_manager.cc:1398] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:13:26.285254 32627 tablet_bootstrap.cc:492] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:26.286746 32627 tablet_bootstrap.cc:654] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:26.286979 32474 raft_consensus.cc:504] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:26.288321 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9000ce76eb154434a88b52e357497767" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:26.288502 31582 raft_consensus.cc:3037] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:26.289050 32627 tablet_bootstrap.cc:492] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:26.289207 32627 ts_tablet_manager.cc:1367] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:13:26.289310 32630 tablet_bootstrap.cc:492] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:26.289453 32630 ts_tablet_manager.cc:1367] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240503 02:13:26.289451 31324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9000ce76eb154434a88b52e357497767" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
I20240503 02:13:26.289649 31324 raft_consensus.cc:3037] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:26.289940 32627 raft_consensus.cc:348] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:26.290146 32627 raft_consensus.cc:374] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:26.290066 32630 raft_consensus.cc:348] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:26.290230 32630 raft_consensus.cc:374] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:26.290225 32627 raft_consensus.cc:729] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:26.290304 32630 raft_consensus.cc:729] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:26.290444 32627 consensus_queue.cc:260] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:26.290479 32630 consensus_queue.cc:260] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:26.291006 32627 ts_tablet_manager.cc:1398] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:26.290997 31582 raft_consensus.cc:2454] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 1.
I20240503 02:13:26.291431 31390 leader_election.cc:302] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:26.291741 32271 raft_consensus.cc:2781] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:26.292430 31324 raft_consensus.cc:2454] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 1.
I20240503 02:13:26.293251 32474 leader_election.cc:288] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 election: Requested vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:26.293356 32630 ts_tablet_manager.cc:1398] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:13:26.294034 32271 raft_consensus.cc:686] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Becoming Leader. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Running, Role: LEADER
I20240503 02:13:26.294349 32271 consensus_queue.cc:237] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:26.296913 31048 catalog_manager.cc:5484] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbb7d16e67d94775b2881712419c0577 (127.23.104.195). New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:26.314126 32559 raft_consensus.cc:482] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:26.314258 32559 raft_consensus.cc:504] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:26.314829 32559 leader_election.cc:288] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101), b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:26.315196 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7626c06dc67448f0b65f67b401d62c88" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:26.315214 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7626c06dc67448f0b65f67b401d62c88" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3" is_pre_election: true
I20240503 02:13:26.315367 31454 raft_consensus.cc:2454] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:26.315388 31582 raft_consensus.cc:2454] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:26.315768 31257 leader_election.cc:302] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:26.316048 32559 raft_consensus.cc:2781] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:26.316175 32559 raft_consensus.cc:482] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:26.316277 32559 raft_consensus.cc:3037] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:26.318141 32559 raft_consensus.cc:504] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:26.318683 32559 leader_election.cc:288] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Requested vote from peers fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101), b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:26.319052 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7626c06dc67448f0b65f67b401d62c88" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:26.319155 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7626c06dc67448f0b65f67b401d62c88" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:26.319229 31454 raft_consensus.cc:3037] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:26.319356 31582 raft_consensus.cc:3037] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:26.321229 31582 raft_consensus.cc:2454] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:26.321239 31454 raft_consensus.cc:2454] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:26.321604 31260 leader_election.cc:302] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:26.321851 32559 raft_consensus.cc:2781] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:26.321965 32559 raft_consensus.cc:686] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Leader. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:26.322227 32559 consensus_queue.cc:237] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:26.324728 31048 catalog_manager.cc:5484] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:26.678877 32271 raft_consensus.cc:482] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:26.679067 32271 raft_consensus.cc:504] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:26.679600 32271 leader_election.cc:288] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:26.680080 31324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e7774779e47ba9d426ac00ad8be62" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" is_pre_election: true
I20240503 02:13:26.680100 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e7774779e47ba9d426ac00ad8be62" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:26.680296 31324 raft_consensus.cc:2454] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 0.
I20240503 02:13:26.680306 31193 raft_consensus.cc:2454] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 0.
I20240503 02:13:26.680749 31389 leader_election.cc:302] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:26.681140 32271 raft_consensus.cc:2781] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:26.681267 32271 raft_consensus.cc:482] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:26.681389 32271 raft_consensus.cc:3037] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:26.683894 32271 raft_consensus.cc:504] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:26.684322 32271 leader_election.cc:288] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 election: Requested vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:26.684655 31324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e7774779e47ba9d426ac00ad8be62" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
I20240503 02:13:26.684727 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e7774779e47ba9d426ac00ad8be62" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:26.684849 31324 raft_consensus.cc:3037] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:26.684896 31193 raft_consensus.cc:3037] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:26.687489 31193 raft_consensus.cc:2454] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 1.
I20240503 02:13:26.687536 31324 raft_consensus.cc:2454] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 1.
I20240503 02:13:26.687876 31389 leader_election.cc:302] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:26.688170 32271 raft_consensus.cc:2781] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:26.688309 32271 raft_consensus.cc:686] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Becoming Leader. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Running, Role: LEADER
I20240503 02:13:26.688555 32271 consensus_queue.cc:237] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:26.691771 31048 catalog_manager.cc:5484] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbb7d16e67d94775b2881712419c0577 (127.23.104.195). New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:26.718494 32559 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:26.743077 32271 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:26.747372 32474 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:26.753021 32651 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:27.128718 32653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:27.129088 32653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:27.137341 32653 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 02:13:27.205644 32271 consensus_queue.cc:1035] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:27.209964 32640 consensus_queue.cc:1035] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 237.121 ms
time per row: 0.118561 ms
I20240503 02:13:27.513445 31051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39754:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:27.521916 31563 tablet_service.cc:1467] Processing CreateTablet for tablet 6ce56ca4030a445db890303244caf308 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ad6bdc3b37e64f27824a7eacb47ca790]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:27.522063 31172 tablet_service.cc:1467] Processing CreateTablet for tablet 6ce56ca4030a445db890303244caf308 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ad6bdc3b37e64f27824a7eacb47ca790]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:27.522382 31563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ce56ca4030a445db890303244caf308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:27.522466 31172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ce56ca4030a445db890303244caf308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:27.522944 31564 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce9e7d46a6e475faaffb1ee26973db7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ad6bdc3b37e64f27824a7eacb47ca790]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:27.523304 31564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce9e7d46a6e475faaffb1ee26973db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:27.524291 31304 tablet_service.cc:1467] Processing CreateTablet for tablet 6ce56ca4030a445db890303244caf308 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ad6bdc3b37e64f27824a7eacb47ca790]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:27.524677 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ce56ca4030a445db890303244caf308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:27.524765 31562 tablet_service.cc:1467] Processing CreateTablet for tablet 9b909a66859144f7b4b649d505c9645e (DEFAULT_TABLE table=rebalance_test_table_6 [id=ad6bdc3b37e64f27824a7eacb47ca790]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:27.525103 31562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b909a66859144f7b4b649d505c9645e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:27.527683 31302 tablet_service.cc:1467] Processing CreateTablet for tablet 9b909a66859144f7b4b649d505c9645e (DEFAULT_TABLE table=rebalance_test_table_6 [id=ad6bdc3b37e64f27824a7eacb47ca790]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:27.528085 31302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b909a66859144f7b4b649d505c9645e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:27.528174 31432 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce9e7d46a6e475faaffb1ee26973db7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ad6bdc3b37e64f27824a7eacb47ca790]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:27.528594 31432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce9e7d46a6e475faaffb1ee26973db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:27.531525 31171 tablet_service.cc:1467] Processing CreateTablet for tablet 9b909a66859144f7b4b649d505c9645e (DEFAULT_TABLE table=rebalance_test_table_6 [id=ad6bdc3b37e64f27824a7eacb47ca790]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:27.531683 31173 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce9e7d46a6e475faaffb1ee26973db7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ad6bdc3b37e64f27824a7eacb47ca790]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:27.531950 31171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b909a66859144f7b4b649d505c9645e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:27.532150 31173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce9e7d46a6e475faaffb1ee26973db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:27.549541 32717 tablet_bootstrap.cc:492] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:27.551837 32717 tablet_bootstrap.cc:654] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:27.553676 32715 tablet_bootstrap.cc:492] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:27.555308 32715 tablet_bootstrap.cc:654] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:27.555451 32714 tablet_bootstrap.cc:492] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:27.556546 32717 tablet_bootstrap.cc:492] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:27.556685 32717 ts_tablet_manager.cc:1367] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240503 02:13:27.557268 32714 tablet_bootstrap.cc:654] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:27.557328 32717 raft_consensus.cc:348] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:27.557502 32717 raft_consensus.cc:374] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:27.557600 32717 raft_consensus.cc:729] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:27.557883 32717 consensus_queue.cc:260] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:27.558720 32717 ts_tablet_manager.cc:1398] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:27.562696 32715 tablet_bootstrap.cc:492] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:27.562903 32715 ts_tablet_manager.cc:1367] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240503 02:13:27.563962 32715 raft_consensus.cc:348] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.564122 32715 raft_consensus.cc:374] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:27.564245 32715 raft_consensus.cc:729] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:27.564460 32715 consensus_queue.cc:260] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.565050 32715 ts_tablet_manager.cc:1398] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:27.565325 32715 tablet_bootstrap.cc:492] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:27.566893 32715 tablet_bootstrap.cc:654] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:27.573918 32715 tablet_bootstrap.cc:492] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:27.574100 32715 ts_tablet_manager.cc:1367] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240503 02:13:27.574622 32715 raft_consensus.cc:348] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.574782 32715 raft_consensus.cc:374] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:27.574873 32715 raft_consensus.cc:729] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:27.575079 32715 consensus_queue.cc:260] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.575568 32715 ts_tablet_manager.cc:1398] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:13:27.579586 32716 tablet_bootstrap.cc:492] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:27.581720 32716 tablet_bootstrap.cc:654] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:27.583817 32714 tablet_bootstrap.cc:492] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:27.583946 32714 ts_tablet_manager.cc:1367] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.009s
I20240503 02:13:27.584573 32716 tablet_bootstrap.cc:492] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:27.584729 32716 ts_tablet_manager.cc:1367] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 02:13:27.584638 32714 raft_consensus.cc:348] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.584815 32714 raft_consensus.cc:374] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:27.584934 32714 raft_consensus.cc:729] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:27.585176 32714 consensus_queue.cc:260] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.585494 32716 raft_consensus.cc:348] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.585734 32714 ts_tablet_manager.cc:1398] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:27.585749 32716 raft_consensus.cc:374] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:27.586066 32714 tablet_bootstrap.cc:492] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:27.587533 32716 raft_consensus.cc:729] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:27.587589 32714 tablet_bootstrap.cc:654] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:27.587889 32716 consensus_queue.cc:260] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.588567 32716 ts_tablet_manager.cc:1398] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:13:27.589628 32716 tablet_bootstrap.cc:492] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:27.594498 32716 tablet_bootstrap.cc:654] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:27.596175 32716 tablet_bootstrap.cc:492] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:27.596328 32716 ts_tablet_manager.cc:1367] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240503 02:13:27.596905 32716 raft_consensus.cc:348] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:27.597107 32716 raft_consensus.cc:374] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:27.597189 32716 raft_consensus.cc:729] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:27.597409 32716 consensus_queue.cc:260] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:27.598011 32716 ts_tablet_manager.cc:1398] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:27.598430 32716 tablet_bootstrap.cc:492] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:27.599413 32714 tablet_bootstrap.cc:492] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:27.599550 32714 ts_tablet_manager.cc:1367] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20240503 02:13:27.600255 32716 tablet_bootstrap.cc:654] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:27.600239 32714 raft_consensus.cc:348] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.600430 32714 raft_consensus.cc:374] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:27.600486 32714 raft_consensus.cc:729] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:27.600682 32714 consensus_queue.cc:260] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.601207 32714 ts_tablet_manager.cc:1398] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:27.601564 32714 tablet_bootstrap.cc:492] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:27.602978 32714 tablet_bootstrap.cc:654] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:27.605144 32714 tablet_bootstrap.cc:492] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:27.605270 32714 ts_tablet_manager.cc:1367] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 02:13:27.605948 32714 raft_consensus.cc:348] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:27.606096 32714 raft_consensus.cc:374] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:27.606185 32714 raft_consensus.cc:729] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:27.606385 32714 consensus_queue.cc:260] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:27.606859 32714 ts_tablet_manager.cc:1398] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:13:27.608606 32693 raft_consensus.cc:482] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:27.608772 32693 raft_consensus.cc:504] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.609484 32693 leader_election.cc:288] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:27.609911 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b909a66859144f7b4b649d505c9645e" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:27.609910 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b909a66859144f7b4b649d505c9645e" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3" is_pre_election: true
I20240503 02:13:27.610080 31193 raft_consensus.cc:2454] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:27.610134 31583 raft_consensus.cc:2454] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:27.610502 31259 leader_election.cc:302] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, b92ff49cec804b1f801c11b64e3259b3, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:27.610817 32693 raft_consensus.cc:2781] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:27.610935 32693 raft_consensus.cc:482] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:27.611024 32693 raft_consensus.cc:3037] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:27.612766 32693 raft_consensus.cc:504] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.613233 32693 leader_election.cc:288] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Requested vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:27.613755 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b909a66859144f7b4b649d505c9645e" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:27.613943 31193 raft_consensus.cc:3037] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:27.614517 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b909a66859144f7b4b649d505c9645e" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:27.614692 31583 raft_consensus.cc:3037] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:27.616546 31583 raft_consensus.cc:2454] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:27.616865 31260 leader_election.cc:302] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:27.617159 32693 raft_consensus.cc:2781] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:27.617316 32693 raft_consensus.cc:686] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Leader. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:27.617571 32693 consensus_queue.cc:237] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.620098 31048 catalog_manager.cc:5484] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:27.624325 32716 tablet_bootstrap.cc:492] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:27.624492 32716 ts_tablet_manager.cc:1367] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
I20240503 02:13:27.625319 32716 raft_consensus.cc:348] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.625522 32716 raft_consensus.cc:374] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:27.625625 32716 raft_consensus.cc:729] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:27.625847 32716 consensus_queue.cc:260] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.626348 32716 ts_tablet_manager.cc:1398] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:27.626952 31193 raft_consensus.cc:2454] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:27.665935 32693 raft_consensus.cc:482] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:27.666169 32693 raft_consensus.cc:504] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.666813 32693 leader_election.cc:288] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:27.667259 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce56ca4030a445db890303244caf308" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3" is_pre_election: true
I20240503 02:13:27.667346 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce56ca4030a445db890303244caf308" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:27.667500 31583 raft_consensus.cc:2454] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:27.667578 31193 raft_consensus.cc:2454] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:27.667915 31260 leader_election.cc:302] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:27.668215 32693 raft_consensus.cc:2781] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:27.668309 32693 raft_consensus.cc:482] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:27.668412 32693 raft_consensus.cc:3037] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:27.670221 32693 raft_consensus.cc:504] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.670802 32693 leader_election.cc:288] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Requested vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:27.671212 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce56ca4030a445db890303244caf308" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:27.671325 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce56ca4030a445db890303244caf308" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:27.671407 31583 raft_consensus.cc:3037] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:27.671555 31193 raft_consensus.cc:3037] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:27.674255 31583 raft_consensus.cc:2454] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:27.674311 31193 raft_consensus.cc:2454] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:27.674671 31259 leader_election.cc:302] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:27.674926 32693 raft_consensus.cc:2781] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:27.675073 32693 raft_consensus.cc:686] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Leader. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:27.675343 32693 consensus_queue.cc:237] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:27.678006 31049 catalog_manager.cc:5484] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:27.716168 32678 raft_consensus.cc:482] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:27.716348 32678 raft_consensus.cc:504] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:27.717800 32678 leader_election.cc:288] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:27.727490 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce9e7d46a6e475faaffb1ee26973db7" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:27.727707 31454 raft_consensus.cc:2454] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 0.
I20240503 02:13:27.728241 31126 leader_election.cc:302] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:27.728432 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce9e7d46a6e475faaffb1ee26973db7" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3" is_pre_election: true
I20240503 02:13:27.728626 31583 raft_consensus.cc:2454] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 0.
I20240503 02:13:27.728694 32678 raft_consensus.cc:2781] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:27.728803 32678 raft_consensus.cc:482] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:27.728899 32678 raft_consensus.cc:3037] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:27.731015 32678 raft_consensus.cc:504] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:27.731624 32678 leader_election.cc:288] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 1 election: Requested vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:27.731923 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce9e7d46a6e475faaffb1ee26973db7" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:27.732028 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce9e7d46a6e475faaffb1ee26973db7" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:27.732151 31583 raft_consensus.cc:3037] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:27.732199 31454 raft_consensus.cc:3037] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:27.734867 31583 raft_consensus.cc:2454] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 1.
I20240503 02:13:27.734867 31454 raft_consensus.cc:2454] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 1.
I20240503 02:13:27.735208 31126 leader_election.cc:302] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:27.735446 32678 raft_consensus.cc:2781] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:27.735895 32678 raft_consensus.cc:686] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 LEADER]: Becoming Leader. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Running, Role: LEADER
I20240503 02:13:27.736367 32678 consensus_queue.cc:237] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:27.739660 31049 catalog_manager.cc:5484] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:28.030689 32689 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:28.048168 32693 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:28.056960 32689 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:28.062330 32738 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:28.127218 32734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:28.127511 32734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:28.134673 32734 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 02:13:28.254979 31454 raft_consensus.cc:1261] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:28.256222 32678 consensus_queue.cc:1035] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:28.258910 31584 raft_consensus.cc:1261] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:28.287956 309 consensus_queue.cc:1035] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:28.343034 315 mvcc.cc:205] Tried to move back new op lower bound from 7023421064205262848 to 7023421062091538432. Current Snapshot: MvccSnapshot[applied={T|T < 7023421064205262848}]
INSERT report
rows total: 2000
time total: 226.717 ms
time per row: 0.113358 ms
I20240503 02:13:28.499743 31049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39754:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:28.508090 31432 tablet_service.cc:1467] Processing CreateTablet for tablet c46912d2d4434d4a8c3fa0c3c19ba14b (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a99e03b1a8d47d1a25bbbd747784951]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:28.508395 31171 tablet_service.cc:1467] Processing CreateTablet for tablet c46912d2d4434d4a8c3fa0c3c19ba14b (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a99e03b1a8d47d1a25bbbd747784951]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:28.508582 31432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46912d2d4434d4a8c3fa0c3c19ba14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:28.508852 31171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46912d2d4434d4a8c3fa0c3c19ba14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:28.509056 31564 tablet_service.cc:1467] Processing CreateTablet for tablet 2de731c4261f49088958535b8b7a65e9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a99e03b1a8d47d1a25bbbd747784951]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:28.509460 31564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de731c4261f49088958535b8b7a65e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:28.509394 31302 tablet_service.cc:1467] Processing CreateTablet for tablet c46912d2d4434d4a8c3fa0c3c19ba14b (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a99e03b1a8d47d1a25bbbd747784951]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:28.509775 31302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46912d2d4434d4a8c3fa0c3c19ba14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:28.512708 31433 tablet_service.cc:1467] Processing CreateTablet for tablet 2de731c4261f49088958535b8b7a65e9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a99e03b1a8d47d1a25bbbd747784951]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:28.513044 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de731c4261f49088958535b8b7a65e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:28.513597 31304 tablet_service.cc:1467] Processing CreateTablet for tablet cb7e8fe39c024108be24bc4c144b247b (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a99e03b1a8d47d1a25bbbd747784951]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:28.513949 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7e8fe39c024108be24bc4c144b247b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:28.516194 31173 tablet_service.cc:1467] Processing CreateTablet for tablet 2de731c4261f49088958535b8b7a65e9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a99e03b1a8d47d1a25bbbd747784951]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:28.516566 31173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de731c4261f49088958535b8b7a65e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:28.516573 31563 tablet_service.cc:1467] Processing CreateTablet for tablet cb7e8fe39c024108be24bc4c144b247b (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a99e03b1a8d47d1a25bbbd747784951]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:28.516956 31563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7e8fe39c024108be24bc4c144b247b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:28.520124 31434 tablet_service.cc:1467] Processing CreateTablet for tablet cb7e8fe39c024108be24bc4c144b247b (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a99e03b1a8d47d1a25bbbd747784951]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:28.520480 31434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7e8fe39c024108be24bc4c144b247b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:28.527918 337 tablet_bootstrap.cc:492] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:28.531981 337 tablet_bootstrap.cc:654] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:28.532720 336 tablet_bootstrap.cc:492] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:28.533560 335 tablet_bootstrap.cc:492] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:28.537207 335 tablet_bootstrap.cc:654] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:28.537266 334 tablet_bootstrap.cc:492] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:28.539230 336 tablet_bootstrap.cc:654] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:28.540632 337 tablet_bootstrap.cc:492] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:28.540875 337 ts_tablet_manager.cc:1367] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240503 02:13:28.541983 337 raft_consensus.cc:348] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:28.542392 337 raft_consensus.cc:374] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:28.542558 337 raft_consensus.cc:729] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:28.542930 337 consensus_queue.cc:260] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:28.543697 337 ts_tablet_manager.cc:1398] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:28.544032 337 tablet_bootstrap.cc:492] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:28.545151 335 tablet_bootstrap.cc:492] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:28.545360 335 ts_tablet_manager.cc:1367] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240503 02:13:28.546451 337 tablet_bootstrap.cc:654] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:28.547700 336 tablet_bootstrap.cc:492] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:28.547945 336 ts_tablet_manager.cc:1367] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.005s
I20240503 02:13:28.546859 335 raft_consensus.cc:348] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:28.548244 335 raft_consensus.cc:374] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:28.548420 335 raft_consensus.cc:729] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:28.548761 335 consensus_queue.cc:260] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:28.548844 336 raft_consensus.cc:348] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:28.549144 336 raft_consensus.cc:374] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:28.549309 336 raft_consensus.cc:729] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:28.549702 335 ts_tablet_manager.cc:1398] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:13:28.549612 336 consensus_queue.cc:260] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:28.550758 335 tablet_bootstrap.cc:492] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:28.550985 336 ts_tablet_manager.cc:1398] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:28.551301 336 tablet_bootstrap.cc:492] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:28.552604 335 tablet_bootstrap.cc:654] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:28.552887 337 tablet_bootstrap.cc:492] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:28.552937 336 tablet_bootstrap.cc:654] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:28.553094 337 ts_tablet_manager.cc:1367] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20240503 02:13:28.557031 337 raft_consensus.cc:348] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:28.557291 337 raft_consensus.cc:374] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:28.557389 337 raft_consensus.cc:729] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:28.557678 337 consensus_queue.cc:260] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:28.558347 337 ts_tablet_manager.cc:1398] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 02:13:28.558738 337 tablet_bootstrap.cc:492] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:28.547966 334 tablet_bootstrap.cc:654] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:28.560742 337 tablet_bootstrap.cc:654] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:28.562202 336 tablet_bootstrap.cc:492] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:28.562412 336 ts_tablet_manager.cc:1367] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240503 02:13:28.563196 336 raft_consensus.cc:348] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:28.563488 336 raft_consensus.cc:374] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:28.563613 336 raft_consensus.cc:729] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:28.564114 336 consensus_queue.cc:260] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:28.564874 336 ts_tablet_manager.cc:1398] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:28.565537 335 tablet_bootstrap.cc:492] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:28.565763 335 ts_tablet_manager.cc:1367] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240503 02:13:28.566764 335 raft_consensus.cc:348] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:28.567059 335 raft_consensus.cc:374] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:28.567178 335 raft_consensus.cc:729] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:28.568302 334 tablet_bootstrap.cc:492] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:28.568449 334 ts_tablet_manager.cc:1367] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.002s
I20240503 02:13:28.568581 335 consensus_queue.cc:260] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:28.569164 334 raft_consensus.cc:348] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:28.569269 335 ts_tablet_manager.cc:1398] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:28.569391 334 raft_consensus.cc:374] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:28.569492 334 raft_consensus.cc:729] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:28.569909 334 consensus_queue.cc:260] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:28.571592 337 tablet_bootstrap.cc:492] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:28.571750 337 ts_tablet_manager.cc:1367] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20240503 02:13:28.572532 337 raft_consensus.cc:348] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:28.572736 337 raft_consensus.cc:374] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:28.572845 337 raft_consensus.cc:729] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:28.573064 337 consensus_queue.cc:260] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:28.573273 334 ts_tablet_manager.cc:1398] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:13:28.573621 337 ts_tablet_manager.cc:1398] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:28.575106 334 tablet_bootstrap.cc:492] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:28.576774 334 tablet_bootstrap.cc:654] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:28.578563 334 tablet_bootstrap.cc:492] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:28.578744 334 ts_tablet_manager.cc:1367] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:28.579327 334 raft_consensus.cc:348] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:28.579501 334 raft_consensus.cc:374] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:28.579588 334 raft_consensus.cc:729] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:28.579851 334 consensus_queue.cc:260] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:28.580361 334 ts_tablet_manager.cc:1398] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:28.589113 32743 raft_consensus.cc:482] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:28.589248 32743 raft_consensus.cc:504] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:28.589809 32743 leader_election.cc:288] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101), b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:28.590101 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7e8fe39c024108be24bc4c144b247b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:28.590286 31454 raft_consensus.cc:2454] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:28.590240 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7e8fe39c024108be24bc4c144b247b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3" is_pre_election: true
I20240503 02:13:28.590452 31583 raft_consensus.cc:2454] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:28.590618 31257 leader_election.cc:302] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:28.590876 32743 raft_consensus.cc:2781] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:28.590996 32743 raft_consensus.cc:482] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:28.591073 32743 raft_consensus.cc:3037] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:28.593806 32743 raft_consensus.cc:504] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:28.594539 32743 leader_election.cc:288] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Requested vote from peers fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101), b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:28.594666 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7e8fe39c024108be24bc4c144b247b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:28.594812 31454 raft_consensus.cc:3037] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:28.595008 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7e8fe39c024108be24bc4c144b247b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:28.595172 31583 raft_consensus.cc:3037] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:28.596817 31454 raft_consensus.cc:2454] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:28.597050 31583 raft_consensus.cc:2454] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:28.597178 31257 leader_election.cc:302] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:28.597455 32743 raft_consensus.cc:2781] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:28.597568 32743 raft_consensus.cc:686] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Leader. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:28.597837 32743 consensus_queue.cc:237] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:28.600135 31047 catalog_manager.cc:5484] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:28.623263 32576 raft_consensus.cc:482] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:28.623400 32576 raft_consensus.cc:504] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:28.624020 32576 leader_election.cc:288] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:28.624413 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de731c4261f49088958535b8b7a65e9" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:28.624529 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de731c4261f49088958535b8b7a65e9" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:28.624651 31454 raft_consensus.cc:2454] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 0.
I20240503 02:13:28.624751 31193 raft_consensus.cc:2454] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 0.
I20240503 02:13:28.625070 31517 leader_election.cc:302] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:28.625332 32576 raft_consensus.cc:2781] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:28.625443 32576 raft_consensus.cc:482] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:28.625514 32576 raft_consensus.cc:3037] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:28.627259 32576 raft_consensus.cc:504] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:28.627934 32576 leader_election.cc:288] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Requested vote from peers fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:28.628316 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de731c4261f49088958535b8b7a65e9" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:28.628512 31454 raft_consensus.cc:3037] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:28.628502 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de731c4261f49088958535b8b7a65e9" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:28.628701 31193 raft_consensus.cc:3037] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:28.630849 31454 raft_consensus.cc:2454] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 1.
I20240503 02:13:28.631050 31193 raft_consensus.cc:2454] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 1.
I20240503 02:13:28.631227 31517 leader_election.cc:302] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:28.637248 348 raft_consensus.cc:2781] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:28.637399 348 raft_consensus.cc:686] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Becoming Leader. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Running, Role: LEADER
I20240503 02:13:28.637756 348 consensus_queue.cc:237] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } }
I20240503 02:13:28.640635 31047 catalog_manager.cc:5484] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196). New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:28.738498 32743 raft_consensus.cc:482] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:28.738732 32743 raft_consensus.cc:504] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:28.739429 32743 leader_election.cc:288] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:28.739753 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46912d2d4434d4a8c3fa0c3c19ba14b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:28.740031 31454 raft_consensus.cc:2454] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:28.740060 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46912d2d4434d4a8c3fa0c3c19ba14b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:28.740319 31193 raft_consensus.cc:2454] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:28.740387 31257 leader_election.cc:302] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:28.740693 32743 raft_consensus.cc:2781] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:28.740795 32743 raft_consensus.cc:482] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:28.740895 32743 raft_consensus.cc:3037] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:28.742770 32743 raft_consensus.cc:504] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:28.743268 32743 leader_election.cc:288] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Requested vote from peers fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:28.743618 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46912d2d4434d4a8c3fa0c3c19ba14b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:28.743628 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46912d2d4434d4a8c3fa0c3c19ba14b" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:28.743832 31454 raft_consensus.cc:3037] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:28.743846 31193 raft_consensus.cc:3037] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:28.746544 31193 raft_consensus.cc:2454] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:28.746546 31454 raft_consensus.cc:2454] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:28.746908 31257 leader_election.cc:302] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:28.747189 32743 raft_consensus.cc:2781] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:28.747328 32743 raft_consensus.cc:686] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Leader. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:28.747530 32743 consensus_queue.cc:237] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:28.750046 31047 catalog_manager.cc:5484] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:29.032238 32738 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:29.040299 32743 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:29.090554 351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:29.090848 351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:29.098064 351 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 02:13:29.126744 32738 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:29.130896 32738 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:29.233450 32576 consensus_queue.cc:1035] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:29.249912 32576 consensus_queue.cc:1035] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:29.358006 31257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:39101, user_credentials={real_user=slave}} blocked reactor thread for 118515us
INSERT report
rows total: 2000
time total: 310.741 ms
time per row: 0.155371 ms
I20240503 02:13:29.558509 31047 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39754:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:29.566694 31171 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8353133ccd4601a3761652546a33b5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=dc947496e186485092d5d621503a4e6f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:29.567203 31171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8353133ccd4601a3761652546a33b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.568011 31563 tablet_service.cc:1467] Processing CreateTablet for tablet 7230f1d648c34351aed29f98795150a2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=dc947496e186485092d5d621503a4e6f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:29.568374 31563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7230f1d648c34351aed29f98795150a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.568327 31304 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8353133ccd4601a3761652546a33b5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=dc947496e186485092d5d621503a4e6f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:29.568704 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8353133ccd4601a3761652546a33b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.572152 31564 tablet_service.cc:1467] Processing CreateTablet for tablet f52b9a3c14a04815800753bda9505cee (DEFAULT_TABLE table=rebalance_test_table_8 [id=dc947496e186485092d5d621503a4e6f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:29.572522 31564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f52b9a3c14a04815800753bda9505cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.572995 418 tablet_bootstrap.cc:492] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:29.573017 31302 tablet_service.cc:1467] Processing CreateTablet for tablet f52b9a3c14a04815800753bda9505cee (DEFAULT_TABLE table=rebalance_test_table_8 [id=dc947496e186485092d5d621503a4e6f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:29.573367 31302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f52b9a3c14a04815800753bda9505cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.575727 418 tablet_bootstrap.cc:654] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:29.577217 31173 tablet_service.cc:1467] Processing CreateTablet for tablet f52b9a3c14a04815800753bda9505cee (DEFAULT_TABLE table=rebalance_test_table_8 [id=dc947496e186485092d5d621503a4e6f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:29.577647 31173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f52b9a3c14a04815800753bda9505cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.580238 31172 tablet_service.cc:1467] Processing CreateTablet for tablet 7230f1d648c34351aed29f98795150a2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=dc947496e186485092d5d621503a4e6f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:29.580683 31172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7230f1d648c34351aed29f98795150a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.581596 31433 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8353133ccd4601a3761652546a33b5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=dc947496e186485092d5d621503a4e6f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:29.582012 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8353133ccd4601a3761652546a33b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.582633 418 tablet_bootstrap.cc:492] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:29.582754 418 ts_tablet_manager.cc:1367] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240503 02:13:29.583570 418 raft_consensus.cc:348] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.583757 418 raft_consensus.cc:374] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:29.583863 418 raft_consensus.cc:729] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.584136 31432 tablet_service.cc:1467] Processing CreateTablet for tablet 7230f1d648c34351aed29f98795150a2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=dc947496e186485092d5d621503a4e6f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:29.584086 418 consensus_queue.cc:260] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.584450 31432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7230f1d648c34351aed29f98795150a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.584829 418 ts_tablet_manager.cc:1398] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:29.585201 418 tablet_bootstrap.cc:492] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:29.586725 418 tablet_bootstrap.cc:654] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:29.593875 422 tablet_bootstrap.cc:492] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:29.595764 418 tablet_bootstrap.cc:492] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:29.595968 418 ts_tablet_manager.cc:1367] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20240503 02:13:29.596253 422 tablet_bootstrap.cc:654] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:29.596685 418 raft_consensus.cc:348] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:29.596901 418 raft_consensus.cc:374] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:29.596982 418 raft_consensus.cc:729] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.597271 418 consensus_queue.cc:260] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:29.597874 418 ts_tablet_manager.cc:1398] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:29.604943 422 tablet_bootstrap.cc:492] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:29.605182 422 ts_tablet_manager.cc:1367] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240503 02:13:29.606184 422 raft_consensus.cc:348] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.606460 422 raft_consensus.cc:374] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:29.606603 422 raft_consensus.cc:729] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.606884 422 consensus_queue.cc:260] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.607661 422 ts_tablet_manager.cc:1398] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:29.608004 422 tablet_bootstrap.cc:492] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:29.608285 424 tablet_bootstrap.cc:492] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:29.610285 422 tablet_bootstrap.cc:654] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:29.610666 424 tablet_bootstrap.cc:654] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:29.612056 419 tablet_bootstrap.cc:492] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:29.613651 419 tablet_bootstrap.cc:654] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:29.615649 419 tablet_bootstrap.cc:492] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:29.615959 419 ts_tablet_manager.cc:1367] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240503 02:13:29.616989 419 raft_consensus.cc:348] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.617305 419 raft_consensus.cc:374] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:29.617449 419 raft_consensus.cc:729] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.617717 419 consensus_queue.cc:260] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.618396 419 ts_tablet_manager.cc:1398] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:29.618477 424 tablet_bootstrap.cc:492] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:29.618734 419 tablet_bootstrap.cc:492] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:29.619078 422 tablet_bootstrap.cc:492] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:29.619237 422 ts_tablet_manager.cc:1367] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240503 02:13:29.619907 424 ts_tablet_manager.cc:1367] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20240503 02:13:29.620246 419 tablet_bootstrap.cc:654] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:29.620760 424 raft_consensus.cc:348] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.621223 422 raft_consensus.cc:348] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.621526 422 raft_consensus.cc:374] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:29.621675 424 raft_consensus.cc:374] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:29.621819 424 raft_consensus.cc:729] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.621959 422 raft_consensus.cc:729] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.625046 424 consensus_queue.cc:260] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.622272 422 consensus_queue.cc:260] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.625501 419 tablet_bootstrap.cc:492] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:29.625675 419 ts_tablet_manager.cc:1367] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 02:13:29.625789 422 ts_tablet_manager.cc:1398] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:13:29.626387 419 raft_consensus.cc:348] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:29.626585 419 raft_consensus.cc:374] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:29.626672 419 raft_consensus.cc:729] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.627007 424 ts_tablet_manager.cc:1398] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 02:13:29.626904 419 consensus_queue.cc:260] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:29.627290 424 tablet_bootstrap.cc:492] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:29.627507 419 ts_tablet_manager.cc:1398] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:29.627652 422 tablet_bootstrap.cc:492] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:29.631381 424 tablet_bootstrap.cc:654] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:29.631769 422 tablet_bootstrap.cc:654] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:29.634785 424 tablet_bootstrap.cc:492] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:29.634922 422 tablet_bootstrap.cc:492] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:29.634946 424 ts_tablet_manager.cc:1367] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 02:13:29.635056 422 ts_tablet_manager.cc:1367] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240503 02:13:29.635517 424 raft_consensus.cc:348] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.635569 422 raft_consensus.cc:348] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:29.635674 424 raft_consensus.cc:374] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:29.635740 422 raft_consensus.cc:374] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:29.635735 424 raft_consensus.cc:729] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.635847 422 raft_consensus.cc:729] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.636032 424 consensus_queue.cc:260] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.636045 422 consensus_queue.cc:260] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:29.636483 424 ts_tablet_manager.cc:1398] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 02:13:29.636979 422 ts_tablet_manager.cc:1398] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:29.669075 401 raft_consensus.cc:482] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:29.669220 401 raft_consensus.cc:504] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.669777 401 leader_election.cc:288] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:29.670122 31191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8353133ccd4601a3761652546a33b5" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:29.670131 31453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8353133ccd4601a3761652546a33b5" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:29.670297 31453 raft_consensus.cc:2454] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:29.670290 31191 raft_consensus.cc:2454] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:29.670634 31257 leader_election.cc:302] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:29.670876 401 raft_consensus.cc:2781] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:29.670970 401 raft_consensus.cc:482] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:29.671056 401 raft_consensus.cc:3037] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:29.672909 401 raft_consensus.cc:504] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.673398 401 leader_election.cc:288] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Requested vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:29.673709 31191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8353133ccd4601a3761652546a33b5" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:29.673767 31453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8353133ccd4601a3761652546a33b5" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:29.673892 31191 raft_consensus.cc:3037] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:29.673907 31453 raft_consensus.cc:3037] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:29.675557 31191 raft_consensus.cc:2454] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:29.675601 31453 raft_consensus.cc:2454] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:29.676024 31259 leader_election.cc:302] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:29.676395 401 raft_consensus.cc:2781] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:29.676544 401 raft_consensus.cc:686] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Leader. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:29.676829 401 consensus_queue.cc:237] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.678236 32474 raft_consensus.cc:482] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:29.678395 32474 raft_consensus.cc:504] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.678998 32474 leader_election.cc:288] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:29.679353 31191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7230f1d648c34351aed29f98795150a2" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" is_pre_election: true
I20240503 02:13:29.679490 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7230f1d648c34351aed29f98795150a2" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3" is_pre_election: true
I20240503 02:13:29.679563 31191 raft_consensus.cc:2454] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 0.
I20240503 02:13:29.679725 31583 raft_consensus.cc:2454] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 0.
I20240503 02:13:29.679941 31389 leader_election.cc:302] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:29.679663 31048 catalog_manager.cc:5484] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:29.680222 32474 raft_consensus.cc:2781] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:29.680289 32474 raft_consensus.cc:482] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:29.680382 32474 raft_consensus.cc:3037] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:29.682384 32474 raft_consensus.cc:504] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.683305 31191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7230f1d648c34351aed29f98795150a2" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:29.683415 32474 leader_election.cc:288] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 election: Requested vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:29.683398 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7230f1d648c34351aed29f98795150a2" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:29.683472 31191 raft_consensus.cc:3037] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:29.683570 31583 raft_consensus.cc:3037] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:29.685318 31191 raft_consensus.cc:2454] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 1.
I20240503 02:13:29.685434 31583 raft_consensus.cc:2454] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 1.
I20240503 02:13:29.685652 31389 leader_election.cc:302] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:29.685878 32474 raft_consensus.cc:2781] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:29.686237 32474 raft_consensus.cc:686] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Becoming Leader. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Running, Role: LEADER
I20240503 02:13:29.686496 309 raft_consensus.cc:482] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:29.686524 32474 consensus_queue.cc:237] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:29.686619 309 raft_consensus.cc:504] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:29.687274 309 leader_election.cc:288] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:29.687573 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f52b9a3c14a04815800753bda9505cee" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3" is_pre_election: true
I20240503 02:13:29.687789 31583 raft_consensus.cc:2454] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 0.
I20240503 02:13:29.688194 31129 leader_election.cc:302] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, b92ff49cec804b1f801c11b64e3259b3; no voters:
I20240503 02:13:29.688498 309 raft_consensus.cc:2781] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:29.688603 309 raft_consensus.cc:482] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:29.688688 309 raft_consensus.cc:3037] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:29.691040 309 raft_consensus.cc:504] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:29.692292 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f52b9a3c14a04815800753bda9505cee" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:29.692472 31583 raft_consensus.cc:3037] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:29.694334 309 leader_election.cc:288] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 1 election: Requested vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:29.696208 31583 raft_consensus.cc:2454] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 1.
I20240503 02:13:29.696580 31129 leader_election.cc:302] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, b92ff49cec804b1f801c11b64e3259b3; no voters:
I20240503 02:13:29.696908 309 raft_consensus.cc:2781] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:29.697310 31324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f52b9a3c14a04815800753bda9505cee" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" is_pre_election: true
I20240503 02:13:29.697386 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f52b9a3c14a04815800753bda9505cee" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
I20240503 02:13:29.697538 31324 raft_consensus.cc:2454] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 0.
I20240503 02:13:29.698239 309 raft_consensus.cc:686] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 LEADER]: Becoming Leader. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Running, Role: LEADER
I20240503 02:13:29.698529 309 consensus_queue.cc:237] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } }
I20240503 02:13:29.700395 31048 catalog_manager.cc:5484] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbb7d16e67d94775b2881712419c0577 (127.23.104.195). New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:29.701269 31046 catalog_manager.cc:5484] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:30.062444 437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:30.062822 437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:30.070822 437 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 02:13:30.176554 401 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:30.189389 401 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:30.230875 32474 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:30.242789 309 consensus_queue.cc:1035] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:30.244918 31324 raft_consensus.cc:3037] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:30.255730 309 consensus_queue.cc:1035] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:30.391690 484 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 295.276 ms
time per row: 0.147638 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:13:31.460443 31047 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39754:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:31.469698 31433 tablet_service.cc:1467] Processing CreateTablet for tablet d9df1587246a4df19fda13f4be2b1177 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3a5935f3f7924b5083fcfc4ee7015893]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:31.469761 31563 tablet_service.cc:1467] Processing CreateTablet for tablet d9df1587246a4df19fda13f4be2b1177 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3a5935f3f7924b5083fcfc4ee7015893]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:31.470144 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9df1587246a4df19fda13f4be2b1177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:31.470160 31563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9df1587246a4df19fda13f4be2b1177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:31.472147 31173 tablet_service.cc:1467] Processing CreateTablet for tablet 4995796bdda64a7cbffa952a05a36673 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3a5935f3f7924b5083fcfc4ee7015893]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:31.472506 31173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4995796bdda64a7cbffa952a05a36673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:31.480433 31302 tablet_service.cc:1467] Processing CreateTablet for tablet b6c71c82cf034bd68aa725e086c56ffb (DEFAULT_TABLE table=rebalance_test_table_9 [id=3a5935f3f7924b5083fcfc4ee7015893]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:31.480852 31302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c71c82cf034bd68aa725e086c56ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:31.480823 31432 tablet_service.cc:1467] Processing CreateTablet for tablet b6c71c82cf034bd68aa725e086c56ffb (DEFAULT_TABLE table=rebalance_test_table_9 [id=3a5935f3f7924b5083fcfc4ee7015893]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:31.481144 31432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c71c82cf034bd68aa725e086c56ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:31.483971 512 tablet_bootstrap.cc:492] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5: Bootstrap starting.
I20240503 02:13:31.484112 31434 tablet_service.cc:1467] Processing CreateTablet for tablet 4995796bdda64a7cbffa952a05a36673 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3a5935f3f7924b5083fcfc4ee7015893]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:31.484401 31434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4995796bdda64a7cbffa952a05a36673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:31.485010 511 tablet_bootstrap.cc:492] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:31.485635 512 tablet_bootstrap.cc:654] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:31.487087 511 tablet_bootstrap.cc:654] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:31.470857 31304 tablet_service.cc:1467] Processing CreateTablet for tablet d9df1587246a4df19fda13f4be2b1177 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3a5935f3f7924b5083fcfc4ee7015893]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:31.488366 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9df1587246a4df19fda13f4be2b1177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:31.489086 31303 tablet_service.cc:1467] Processing CreateTablet for tablet 4995796bdda64a7cbffa952a05a36673 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3a5935f3f7924b5083fcfc4ee7015893]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:31.489574 31303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4995796bdda64a7cbffa952a05a36673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:31.492542 31563 tablet_service.cc:1467] Processing CreateTablet for tablet b6c71c82cf034bd68aa725e086c56ffb (DEFAULT_TABLE table=rebalance_test_table_9 [id=3a5935f3f7924b5083fcfc4ee7015893]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:31.492878 31563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c71c82cf034bd68aa725e086c56ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:31.495880 515 tablet_bootstrap.cc:492] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:31.497419 515 tablet_bootstrap.cc:654] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:31.500475 510 tablet_bootstrap.cc:492] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:31.501593 512 tablet_bootstrap.cc:492] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5: No bootstrap required, opened a new log
I20240503 02:13:31.501787 512 ts_tablet_manager.cc:1367] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240503 02:13:31.501896 511 tablet_bootstrap.cc:492] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:31.502090 511 ts_tablet_manager.cc:1367] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.003s
I20240503 02:13:31.502621 510 tablet_bootstrap.cc:654] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:31.502584 512 raft_consensus.cc:348] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.502895 512 raft_consensus.cc:374] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:31.502846 511 raft_consensus.cc:348] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:31.503046 512 raft_consensus.cc:729] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Initialized, Role: FOLLOWER
I20240503 02:13:31.503131 511 raft_consensus.cc:374] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:31.503261 511 raft_consensus.cc:729] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:31.503330 512 consensus_queue.cc:260] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.503937 512 ts_tablet_manager.cc:1398] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:31.503540 511 consensus_queue.cc:260] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:31.505095 511 ts_tablet_manager.cc:1398] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:13:31.505419 511 tablet_bootstrap.cc:492] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3: Bootstrap starting.
I20240503 02:13:31.507805 511 tablet_bootstrap.cc:654] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:31.509514 515 tablet_bootstrap.cc:492] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:31.509685 515 ts_tablet_manager.cc:1367] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240503 02:13:31.510236 510 tablet_bootstrap.cc:492] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:31.510583 510 ts_tablet_manager.cc:1367] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240503 02:13:31.510951 515 raft_consensus.cc:348] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.511381 515 raft_consensus.cc:374] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:31.511369 510 raft_consensus.cc:348] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:31.511533 510 raft_consensus.cc:374] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:31.511516 515 raft_consensus.cc:729] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:31.511623 510 raft_consensus.cc:729] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:31.511890 510 consensus_queue.cc:260] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:31.511806 515 consensus_queue.cc:260] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.512372 510 ts_tablet_manager.cc:1398] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:31.512461 515 ts_tablet_manager.cc:1398] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:31.512760 515 tablet_bootstrap.cc:492] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:31.512770 510 tablet_bootstrap.cc:492] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:31.514626 515 tablet_bootstrap.cc:654] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:31.515717 510 tablet_bootstrap.cc:654] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:31.516501 511 tablet_bootstrap.cc:492] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3: No bootstrap required, opened a new log
I20240503 02:13:31.516729 511 ts_tablet_manager.cc:1367] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 02:13:31.517711 511 raft_consensus.cc:348] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.518047 511 raft_consensus.cc:374] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:31.518182 511 raft_consensus.cc:729] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Initialized, Role: FOLLOWER
I20240503 02:13:31.518707 511 consensus_queue.cc:260] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.519582 511 ts_tablet_manager.cc:1398] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:31.520833 515 tablet_bootstrap.cc:492] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:31.521034 515 ts_tablet_manager.cc:1367] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20240503 02:13:31.521602 510 tablet_bootstrap.cc:492] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:31.521621 515 raft_consensus.cc:348] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.521795 510 ts_tablet_manager.cc:1367] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20240503 02:13:31.521823 515 raft_consensus.cc:374] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:31.521931 515 raft_consensus.cc:729] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:31.522150 515 consensus_queue.cc:260] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.522507 510 raft_consensus.cc:348] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.522732 515 ts_tablet_manager.cc:1398] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:31.522771 510 raft_consensus.cc:374] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:31.522959 510 raft_consensus.cc:729] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:31.523062 515 tablet_bootstrap.cc:492] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Bootstrap starting.
I20240503 02:13:31.523262 510 consensus_queue.cc:260] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.523830 510 ts_tablet_manager.cc:1398] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:31.524498 515 tablet_bootstrap.cc:654] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:31.526088 510 tablet_bootstrap.cc:492] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577: Bootstrap starting.
I20240503 02:13:31.526134 515 tablet_bootstrap.cc:492] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: No bootstrap required, opened a new log
I20240503 02:13:31.526329 515 ts_tablet_manager.cc:1367] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:31.527042 515 raft_consensus.cc:348] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:31.527375 515 raft_consensus.cc:374] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:31.527513 515 raft_consensus.cc:729] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Initialized, Role: FOLLOWER
I20240503 02:13:31.527817 515 consensus_queue.cc:260] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:31.528280 515 ts_tablet_manager.cc:1398] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:31.528470 510 tablet_bootstrap.cc:654] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:31.531169 510 tablet_bootstrap.cc:492] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577: No bootstrap required, opened a new log
I20240503 02:13:31.531276 510 ts_tablet_manager.cc:1367] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:31.531747 510 raft_consensus.cc:348] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.531945 510 raft_consensus.cc:374] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:31.532018 510 raft_consensus.cc:729] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Initialized, Role: FOLLOWER
I20240503 02:13:31.532204 510 consensus_queue.cc:260] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.532622 510 ts_tablet_manager.cc:1398] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:13:31.767877 523 raft_consensus.cc:482] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:31.768054 523 raft_consensus.cc:504] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.768589 523 leader_election.cc:288] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:31.769027 31453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c71c82cf034bd68aa725e086c56ffb" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:31.769034 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c71c82cf034bd68aa725e086c56ffb" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3" is_pre_election: true
I20240503 02:13:31.769251 31583 raft_consensus.cc:2454] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:31.769258 31453 raft_consensus.cc:2454] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 0.
I20240503 02:13:31.769686 31257 leader_election.cc:302] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:31.769932 523 raft_consensus.cc:2781] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:31.770045 523 raft_consensus.cc:482] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:31.770133 523 raft_consensus.cc:3037] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:31.772553 523 raft_consensus.cc:504] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.772925 491 raft_consensus.cc:482] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:31.773100 523 leader_election.cc:288] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Requested vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:31.773078 491 raft_consensus.cc:504] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:31.773512 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c71c82cf034bd68aa725e086c56ffb" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:31.773587 31453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c71c82cf034bd68aa725e086c56ffb" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:31.773711 31583 raft_consensus.cc:3037] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:31.773808 491 leader_election.cc:288] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:31.774140 31453 raft_consensus.cc:3037] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:31.774389 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df1587246a4df19fda13f4be2b1177" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" is_pre_election: true
I20240503 02:13:31.774590 31323 raft_consensus.cc:2454] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 0.
I20240503 02:13:31.774585 31582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df1587246a4df19fda13f4be2b1177" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3" is_pre_election: true
I20240503 02:13:31.774806 31582 raft_consensus.cc:2454] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 0.
I20240503 02:13:31.775056 31388 leader_election.cc:302] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:31.775318 491 raft_consensus.cc:2781] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:31.775449 491 raft_consensus.cc:482] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:31.775528 491 raft_consensus.cc:3037] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:31.776247 31583 raft_consensus.cc:2454] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:31.776607 31260 leader_election.cc:302] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:31.776862 523 raft_consensus.cc:2781] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:31.777015 523 raft_consensus.cc:686] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Leader. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:31.777194 31453 raft_consensus.cc:2454] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 1.
I20240503 02:13:31.777324 523 consensus_queue.cc:237] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.778129 491 raft_consensus.cc:504] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:31.778939 491 leader_election.cc:288] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 election: Requested vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:31.779316 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df1587246a4df19fda13f4be2b1177" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:31.779467 31583 raft_consensus.cc:3037] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:31.781334 31583 raft_consensus.cc:2454] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 1.
I20240503 02:13:31.781708 31390 leader_election.cc:302] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:31.781810 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df1587246a4df19fda13f4be2b1177" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
I20240503 02:13:31.782021 31323 raft_consensus.cc:3037] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:31.782027 491 raft_consensus.cc:2781] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:31.781853 31047 catalog_manager.cc:5484] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:31.782413 491 raft_consensus.cc:686] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Becoming Leader. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Running, Role: LEADER
I20240503 02:13:31.783190 491 consensus_queue.cc:237] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } }
I20240503 02:13:31.784584 31323 raft_consensus.cc:2454] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 1.
I20240503 02:13:31.788161 31047 catalog_manager.cc:5484] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbb7d16e67d94775b2881712419c0577 (127.23.104.195). New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:31.796437 483 raft_consensus.cc:482] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:31.796586 483 raft_consensus.cc:504] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.797223 483 leader_election.cc:288] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:31.797580 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4995796bdda64a7cbffa952a05a36673" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" is_pre_election: true
I20240503 02:13:31.797674 31453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4995796bdda64a7cbffa952a05a36673" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577" is_pre_election: true
I20240503 02:13:31.797767 31323 raft_consensus.cc:2454] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 0.
I20240503 02:13:31.797875 31453 raft_consensus.cc:2454] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 0.
I20240503 02:13:31.798184 31127 leader_election.cc:302] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:31.798456 483 raft_consensus.cc:2781] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:31.798568 483 raft_consensus.cc:482] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:31.798645 483 raft_consensus.cc:3037] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:31.800493 483 raft_consensus.cc:504] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.801008 483 leader_election.cc:288] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 1 election: Requested vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:31.801340 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4995796bdda64a7cbffa952a05a36673" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
I20240503 02:13:31.801450 31453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4995796bdda64a7cbffa952a05a36673" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:31.801570 31323 raft_consensus.cc:3037] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:31.801635 31453 raft_consensus.cc:3037] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:31.803774 31453 raft_consensus.cc:2454] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 1.
I20240503 02:13:31.803841 31323 raft_consensus.cc:2454] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 1.
I20240503 02:13:31.804239 31126 leader_election.cc:302] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:31.804556 483 raft_consensus.cc:2781] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:31.804924 483 raft_consensus.cc:686] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 LEADER]: Becoming Leader. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Running, Role: LEADER
I20240503 02:13:31.805202 483 consensus_queue.cc:237] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } }
I20240503 02:13:31.808012 31047 catalog_manager.cc:5484] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:32.197717 530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:32.198040 530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:32.205725 530 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:32.209771 483 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:32.219213 483 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240503 02:13:32.268065 491 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:32.273370 525 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:32.380496 400 consensus_queue.cc:1035] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:32.392598 31454 raft_consensus.cc:1261] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:32.393735 400 consensus_queue.cc:1035] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:32.464294 31260 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:39849, user_credentials={real_user=slave}} blocked reactor thread for 46994us
INSERT report
rows total: 2000
time total: 304.042 ms
time per row: 0.152021 ms
I20240503 02:13:32.690238 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.197:37543
--local_ip_for_outbound_sockets=127.23.104.197
--tserver_master_addrs=127.23.104.254:43173
--webserver_port=33139
--webserver_interface=127.23.104.197
--builtin_ntp_servers=127.23.104.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 02:13:33.040990 591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:33.041335 591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:33.041532 591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:33.049757 591 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:33.049877 591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:33.050050 591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.197
I20240503 02:13:33.059099 591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.197:37543
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.23.104.197
--webserver_port=33139
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.23.104.254:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.197
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:33.060498 591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:33.062413 591 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:33.074766 596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:33.075843 597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:33.076403 599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:33.076999 591 server_base.cc:1034] running on GCE node
I20240503 02:13:33.077561 591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:33.078486 591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:33.079691 591 hybrid_clock.cc:648] HybridClock initialized: now 1714702413079634 us; error 55 us; skew 500 ppm
I20240503 02:13:33.080034 591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:33.082166 591 webserver.cc:460] Webserver started at http://127.23.104.197:33139/ using document root <none> and password file <none>
I20240503 02:13:33.082886 591 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:33.083020 591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:33.088382 591 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:13:33.090966 605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:33.092697 591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:33.092871 591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "13d0f62249b34b01a3a47c6afe10e750"
format_stamp: "Formatted at 2024-05-03 02:13:12 on dist-test-slave-38ht"
I20240503 02:13:33.093694 591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:33.114341 591 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:33.115175 591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:33.115453 591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:33.116442 591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:33.117908 591 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:33.117980 591 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:33.118099 591 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:33.118147 591 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:33.151150 591 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.197:37543
I20240503 02:13:33.151185 717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.197:37543 every 8 connection(s)
I20240503 02:13:33.152596 591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:13:33.161443 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 591
I20240503 02:13:33.162134 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.198:38195
--local_ip_for_outbound_sockets=127.23.104.198
--tserver_master_addrs=127.23.104.254:43173
--webserver_port=34573
--webserver_interface=127.23.104.198
--builtin_ntp_servers=127.23.104.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:13:33.165773 718 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43173
I20240503 02:13:33.166075 718 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:33.166884 718 heartbeater.cc:511] Master 127.23.104.254:43173 requested a full tablet report, sending...
I20240503 02:13:33.168463 31047 ts_manager.cc:194] Re-registered known tserver with Master: 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543)
I20240503 02:13:33.169265 31047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.197:38621
W20240503 02:13:33.514175 722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:33.514480 722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:33.514588 722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:33.522083 722 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:33.522199 722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:33.522325 722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.198
I20240503 02:13:33.531708 722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.198:38195
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.23.104.198
--webserver_port=34573
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.23.104.254:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.198
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:33.532968 722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:33.534626 722 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:33.548961 728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:33.549446 727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:33.549891 730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:33.550581 722 server_base.cc:1034] running on GCE node
I20240503 02:13:33.551179 722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:33.552224 722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:33.553483 722 hybrid_clock.cc:648] HybridClock initialized: now 1714702413553408 us; error 46 us; skew 500 ppm
I20240503 02:13:33.554112 722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:33.562155 722 webserver.cc:460] Webserver started at http://127.23.104.198:34573/ using document root <none> and password file <none>
I20240503 02:13:33.562947 722 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:33.563079 722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:33.569044 722 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:13:33.571816 736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:33.573115 722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 02:13:33.573267 722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "05ef0a15decc468ea3a1af8828b0f630"
format_stamp: "Formatted at 2024-05-03 02:13:12 on dist-test-slave-38ht"
I20240503 02:13:33.574056 722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:33.587965 722 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:33.588752 722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:33.589027 722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:33.590003 722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:33.591926 722 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:33.592001 722 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:33.592114 722 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:33.592164 722 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:33.623759 722 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.198:38195
I20240503 02:13:33.623813 848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.198:38195 every 8 connection(s)
I20240503 02:13:33.625205 722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:13:33.633267 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 722
I20240503 02:13:33.633831 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.199:36997
--local_ip_for_outbound_sockets=127.23.104.199
--tserver_master_addrs=127.23.104.254:43173
--webserver_port=35583
--webserver_interface=127.23.104.199
--builtin_ntp_servers=127.23.104.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:13:33.637645 849 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43173
I20240503 02:13:33.637954 849 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:33.638911 849 heartbeater.cc:511] Master 127.23.104.254:43173 requested a full tablet report, sending...
I20240503 02:13:33.640455 31047 ts_manager.cc:194] Re-registered known tserver with Master: 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195)
I20240503 02:13:33.641443 31047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.198:40517
W20240503 02:13:33.989272 852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:33.989552 852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:33.989702 852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:33.997607 852 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:33.997735 852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:33.997903 852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.199
I20240503 02:13:34.008564 852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.199:36997
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.23.104.199
--webserver_port=35583
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.23.104.254:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.199
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:34.009877 852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:34.011673 852 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:34.024286 859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:34.024904 860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:34.025254 862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:34.025907 852 server_base.cc:1034] running on GCE node
I20240503 02:13:34.026459 852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:34.027392 852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:34.028604 852 hybrid_clock.cc:648] HybridClock initialized: now 1714702414028547 us; error 51 us; skew 500 ppm
I20240503 02:13:34.028941 852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:34.031153 852 webserver.cc:460] Webserver started at http://127.23.104.199:35583/ using document root <none> and password file <none>
I20240503 02:13:34.031906 852 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:34.032037 852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:34.037099 852 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:13:34.040056 868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:34.042045 852 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:13:34.042217 852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4a3e3937b61447598fd76bc57f5d0939"
format_stamp: "Formatted at 2024-05-03 02:13:13 on dist-test-slave-38ht"
I20240503 02:13:34.043010 852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:34.059614 852 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:34.060424 852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:34.060716 852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:34.061640 852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:34.063076 852 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:34.063148 852 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:34.063230 852 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:34.063270 852 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:34.093879 852 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.199:36997
I20240503 02:13:34.093952 980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.199:36997 every 8 connection(s)
I20240503 02:13:34.095260 852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:13:34.096556 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 852
I20240503 02:13:34.097200 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.200:37401
--local_ip_for_outbound_sockets=127.23.104.200
--tserver_master_addrs=127.23.104.254:43173
--webserver_port=37389
--webserver_interface=127.23.104.200
--builtin_ntp_servers=127.23.104.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:13:34.110026 981 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43173
I20240503 02:13:34.110406 981 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:34.111254 981 heartbeater.cc:511] Master 127.23.104.254:43173 requested a full tablet report, sending...
I20240503 02:13:34.112674 31047 ts_manager.cc:194] Re-registered known tserver with Master: 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997)
I20240503 02:13:34.113813 31047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.199:57467
I20240503 02:13:34.171979 718 heartbeater.cc:503] Master 127.23.104.254:43173 was elected leader, sending a full tablet report...
W20240503 02:13:34.446136 984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:34.446419 984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:34.446552 984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:34.454560 984 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:34.454675 984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:34.454842 984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.200
I20240503 02:13:34.464641 984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.200:37401
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.23.104.200
--webserver_port=37389
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.23.104.254:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.200
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:34.465940 984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:34.467882 984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:34.480661 992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:34.481164 994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:34.481750 991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:34.482403 984 server_base.cc:1034] running on GCE node
I20240503 02:13:34.482941 984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:34.483871 984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:34.485064 984 hybrid_clock.cc:648] HybridClock initialized: now 1714702414485006 us; error 48 us; skew 500 ppm
I20240503 02:13:34.485380 984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:34.487886 984 webserver.cc:460] Webserver started at http://127.23.104.200:37389/ using document root <none> and password file <none>
I20240503 02:13:34.488708 984 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:34.488829 984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:34.494361 984 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:13:34.497131 1000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:34.498553 984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:34.498683 984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0429a57df6ba4f92a73e779b04b05040"
format_stamp: "Formatted at 2024-05-03 02:13:13 on dist-test-slave-38ht"
I20240503 02:13:34.499506 984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:34.524148 984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:34.524982 984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:34.525308 984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:34.526300 984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:34.527735 984 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:34.527868 984 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:34.528000 984 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:34.528071 984 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:34.571803 984 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.200:37401
I20240503 02:13:34.571888 1112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.200:37401 every 8 connection(s)
I20240503 02:13:34.573235 984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:13:34.581456 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 984
I20240503 02:13:34.586159 1113 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43173
I20240503 02:13:34.586493 1113 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:34.587992 1113 heartbeater.cc:511] Master 127.23.104.254:43173 requested a full tablet report, sending...
I20240503 02:13:34.589597 31047 ts_manager.cc:194] Re-registered known tserver with Master: 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401)
I20240503 02:13:34.590415 31047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.200:36743
I20240503 02:13:34.647238 849 heartbeater.cc:503] Master 127.23.104.254:43173 was elected leader, sending a full tablet report...
W20240503 02:13:34.957535 1118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:34.957881 1118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:34.966002 1118 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:35.116879 981 heartbeater.cc:503] Master 127.23.104.254:43173 was elected leader, sending a full tablet report...
I20240503 02:13:35.226835 1048 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:35.229790 1048 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:35.262008 784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:35.309782 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:35.310654 653 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:35.311069 652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:35.319366 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:35.326419 784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:35.419696 31563 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:35.446305 31173 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:35.446908 31171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:35.449445 31303 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:35.510680 31433 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:35.511390 31563 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:35.544776 31303 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:35.592204 31433 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:35.594794 1113 heartbeater.cc:503] Master 127.23.104.254:43173 was elected leader, sending a full tablet report...
I20240503 02:13:35.909291 784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:35.943542 31171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:35.943511 1048 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:35.961683 652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:35.963605 31433 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:35.964892 31303 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:35.989441 31563 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:36.063344 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:36.317746 1048 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:36.332718 784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:36.351583 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:36.362963 31171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:36.379844 652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:36.408929 31563 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:36.423043 31303 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:36.427853 31433 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:13:36.442492 1256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:36.442993 1256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:36.455298 1256 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:36.566706 31583 consensus_queue.cc:237] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:36.570863 31191 raft_consensus.cc:1261] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:36.571607 1314 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:36.575843 31323 raft_consensus.cc:1261] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:36.575265 1217 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:36.576304 31191 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:36.578984 1314 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:36.583341 31046 catalog_manager.cc:5484] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
W20240503 02:13:36.586350 31518 consensus_peers.cc:485] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 -> Peer 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543): Couldn't send request to peer 13d0f62249b34b01a3a47c6afe10e750. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80a15c3541fc4068acac595bd909f8d8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:36.587316 31324 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:36.612298 31583 consensus_queue.cc:237] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:36.615502 31454 raft_consensus.cc:1261] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:36.616089 31324 raft_consensus.cc:1261] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:36.617548 1217 consensus_queue.cc:1035] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:36.618732 1217 consensus_queue.cc:1035] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:36.622236 1314 raft_consensus.cc:2932] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:36.622926 31324 raft_consensus.cc:2932] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
W20240503 02:13:36.628475 31518 consensus_peers.cc:485] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 -> Peer 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997): Couldn't send request to peer 4a3e3937b61447598fd76bc57f5d0939. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c185751912de4f78808785a620dd9746. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:36.632130 31049 catalog_manager.cc:5484] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:36.656585 31453 raft_consensus.cc:2932] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:36.664002 31583 consensus_queue.cc:237] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:36.667464 31191 raft_consensus.cc:1261] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:36.668429 31453 raft_consensus.cc:1261] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:36.668541 1336 consensus_queue.cc:1035] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:36.669894 1314 consensus_queue.cc:1035] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:36.678599 31520 consensus_peers.cc:485] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 -> Peer 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401): Couldn't send request to peer 0429a57df6ba4f92a73e779b04b05040. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 091475e28bb344e78b1e4e48fa548ef8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:36.686553 1336 raft_consensus.cc:2932] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:36.687436 31192 raft_consensus.cc:2932] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:36.689832 31453 raft_consensus.cc:2932] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:36.693642 31046 catalog_manager.cc:5484] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:36.721776 31453 consensus_queue.cc:237] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:36.725201 31583 raft_consensus.cc:1261] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:36.726197 1294 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:36.728724 31191 raft_consensus.cc:1261] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:36.729527 1294 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:36.744573 1294 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:36.748800 31582 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
W20240503 02:13:36.753115 31389 consensus_peers.cc:485] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 -> Peer 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195): Couldn't send request to peer 05ef0a15decc468ea3a1af8828b0f630. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7230f1d648c34351aed29f98795150a2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:36.755080 31044 catalog_manager.cc:5484] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:36.776547 1048 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:36.776610 784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:36.778286 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:36.778565 31192 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:36.785087 652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:36.796659 31171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:36.801522 31563 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:36.833634 31433 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:36.870944 31303 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240503 02:13:36.897135 31126 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:39101, user_credentials={real_user=slave}} blocked reactor thread for 69066.7us
I20240503 02:13:36.943954 31324 consensus_queue.cc:237] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:36.947389 31582 raft_consensus.cc:1261] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:36.947705 31191 raft_consensus.cc:1261] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:36.948319 1357 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:36.948632 989 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:36.968106 1357 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:36.974031 31583 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:36.974179 31191 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
W20240503 02:13:36.982556 31258 consensus_peers.cc:485] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543): Couldn't send request to peer 13d0f62249b34b01a3a47c6afe10e750. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b909a66859144f7b4b649d505c9645e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:36.983429 31044 catalog_manager.cc:5484] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
Runtime error: ksck discovered errors
I20240503 02:13:37.030913 31324 consensus_queue.cc:237] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:37.041572 1414 ts_tablet_manager.cc:902] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939: Initiating tablet copy from peer b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:37.042964 31583 raft_consensus.cc:1261] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.043452 31191 raft_consensus.cc:1261] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.044127 1357 consensus_queue.cc:1035] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.044724 1402 consensus_queue.cc:1035] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:37.050917 31260 consensus_peers.cc:485] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401): Couldn't send request to peer 0429a57df6ba4f92a73e779b04b05040. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dc576d8e72647c08736ae9508f6d27b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:37.057371 1407 raft_consensus.cc:2932] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:37.060590 31583 raft_consensus.cc:2932] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:37.062198 31191 raft_consensus.cc:2932] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:37.069377 31044 catalog_manager.cc:5484] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:37.109764 1414 tablet_copy_client.cc:308] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:39849
I20240503 02:13:37.114383 31323 consensus_queue.cc:237] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:37.124544 31191 raft_consensus.cc:1261] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.125562 989 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.128177 31454 raft_consensus.cc:1261] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.129142 1407 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:37.131064 31258 consensus_peers.cc:485] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997): Couldn't send request to peer 4a3e3937b61447598fd76bc57f5d0939. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f8353133ccd4601a3761652546a33b5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:37.131969 31604 tablet_copy_service.cc:140] P b92ff49cec804b1f801c11b64e3259b3: Received BeginTabletCopySession request for tablet c185751912de4f78808785a620dd9746 from peer 4a3e3937b61447598fd76bc57f5d0939 ({username='slave'} at 127.23.104.199:40937)
I20240503 02:13:37.132225 31604 tablet_copy_service.cc:161] P b92ff49cec804b1f801c11b64e3259b3: Beginning new tablet copy session on tablet c185751912de4f78808785a620dd9746 from peer 4a3e3937b61447598fd76bc57f5d0939 at {username='slave'} at 127.23.104.199:40937: session id = 4a3e3937b61447598fd76bc57f5d0939-c185751912de4f78808785a620dd9746
I20240503 02:13:37.132278 1407 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:37.136121 31604 tablet_copy_source_session.cc:205] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:37.136049 31454 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:37.141924 1414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c185751912de4f78808785a620dd9746. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:37.142204 31046 catalog_manager.cc:5484] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:37.157318 1414 tablet_copy_client.cc:783] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:37.157681 1414 tablet_copy_client.cc:647] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:37.160323 1424 tablet_copy_client.cc:1002] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:37.170465 31191 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:37.177915 1435 ts_tablet_manager.cc:902] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750: Initiating tablet copy from peer b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:37.181053 31454 consensus_queue.cc:237] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:37.184415 31323 raft_consensus.cc:1261] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.184435 31191 raft_consensus.cc:1261] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:37.184470 31389 consensus_peers.cc:485] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 -> Peer 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195): Couldn't send request to peer 05ef0a15decc468ea3a1af8828b0f630. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37ca6554fec24a07865da3eabef27152. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:37.185101 853 consensus_queue.cc:1035] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.185395 1294 consensus_queue.cc:1035] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.186631 1435 tablet_copy_client.cc:308] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:39849
I20240503 02:13:37.192871 1294 raft_consensus.cc:2932] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:37.196498 31323 raft_consensus.cc:2932] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:37.201148 31192 raft_consensus.cc:2932] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:37.203824 31604 tablet_copy_service.cc:140] P b92ff49cec804b1f801c11b64e3259b3: Received BeginTabletCopySession request for tablet 80a15c3541fc4068acac595bd909f8d8 from peer 13d0f62249b34b01a3a47c6afe10e750 ({username='slave'} at 127.23.104.197:53893)
I20240503 02:13:37.204007 31604 tablet_copy_service.cc:161] P b92ff49cec804b1f801c11b64e3259b3: Beginning new tablet copy session on tablet 80a15c3541fc4068acac595bd909f8d8 from peer 13d0f62249b34b01a3a47c6afe10e750 at {username='slave'} at 127.23.104.197:53893: session id = 13d0f62249b34b01a3a47c6afe10e750-80a15c3541fc4068acac595bd909f8d8
I20240503 02:13:37.206319 31604 tablet_copy_source_session.cc:205] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:37.206509 31048 catalog_manager.cc:5484] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:37.211746 1435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80a15c3541fc4068acac595bd909f8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:37.223112 1435 tablet_copy_client.cc:783] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:37.223482 1435 tablet_copy_client.cc:647] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:37.225939 1440 tablet_copy_client.cc:1002] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:37.237020 31191 consensus_queue.cc:237] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:37.239749 31454 raft_consensus.cc:1261] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.240361 483 consensus_queue.cc:1035] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.243963 1397 raft_consensus.cc:2932] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:37.247105 31454 raft_consensus.cc:2932] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:37.254530 31046 catalog_manager.cc:5484] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:37.261332 31583 raft_consensus.cc:1261] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.262045 1451 consensus_queue.cc:1035] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:37.270443 31129 consensus_peers.cc:485] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 -> Peer 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401): Couldn't send request to peer 0429a57df6ba4f92a73e779b04b05040. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ce9e7d46a6e475faaffb1ee26973db7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:37.277051 1457 ts_tablet_manager.cc:902] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630: Initiating tablet copy from peer fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:37.277863 31583 raft_consensus.cc:2932] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:37.305099 1461 ts_tablet_manager.cc:902] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040: Initiating tablet copy from peer b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:37.305871 1457 tablet_copy_client.cc:308] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:39101
I20240503 02:13:37.318110 31453 consensus_queue.cc:237] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:37.321832 31583 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.322672 1294 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.324693 31323 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.325309 1294 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.326463 853 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:37.327144 31583 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:37.333297 31048 catalog_manager.cc:5484] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:37.349113 31388 consensus_peers.cc:485] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 -> Peer 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543): Couldn't send request to peer 13d0f62249b34b01a3a47c6afe10e750. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9000ce76eb154434a88b52e357497767. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:37.353426 31323 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:37.371053 31474 tablet_copy_service.cc:140] P fbb7d16e67d94775b2881712419c0577: Received BeginTabletCopySession request for tablet 7230f1d648c34351aed29f98795150a2 from peer 05ef0a15decc468ea3a1af8828b0f630 ({username='slave'} at 127.23.104.198:53527)
I20240503 02:13:37.371356 31474 tablet_copy_service.cc:161] P fbb7d16e67d94775b2881712419c0577: Beginning new tablet copy session on tablet 7230f1d648c34351aed29f98795150a2 from peer 05ef0a15decc468ea3a1af8828b0f630 at {username='slave'} at 127.23.104.198:53527: session id = 05ef0a15decc468ea3a1af8828b0f630-7230f1d648c34351aed29f98795150a2
I20240503 02:13:37.376549 31474 tablet_copy_source_session.cc:205] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:37.381930 31323 consensus_queue.cc:237] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:37.382722 1457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7230f1d648c34351aed29f98795150a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:37.392565 31583 raft_consensus.cc:1261] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.393288 989 consensus_queue.cc:1035] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.396347 1461 tablet_copy_client.cc:308] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:39849
I20240503 02:13:37.399194 1457 tablet_copy_client.cc:783] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:37.399674 1457 tablet_copy_client.cc:647] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:37.402204 31453 raft_consensus.cc:1261] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.403671 989 consensus_queue.cc:1035] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.420847 989 raft_consensus.cc:2932] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:37.425110 31454 raft_consensus.cc:2932] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:37.427179 1464 tablet_copy_client.cc:1002] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:37.430166 31044 catalog_manager.cc:5484] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:37.437180 31259 consensus_peers.cc:485] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195): Couldn't send request to peer 05ef0a15decc468ea3a1af8828b0f630. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6c71c82cf034bd68aa725e086c56ffb. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:37.439114 31583 raft_consensus.cc:2932] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:37.453238 31604 tablet_copy_service.cc:140] P b92ff49cec804b1f801c11b64e3259b3: Received BeginTabletCopySession request for tablet 091475e28bb344e78b1e4e48fa548ef8 from peer 0429a57df6ba4f92a73e779b04b05040 ({username='slave'} at 127.23.104.200:59587)
I20240503 02:13:37.453434 31604 tablet_copy_service.cc:161] P b92ff49cec804b1f801c11b64e3259b3: Beginning new tablet copy session on tablet 091475e28bb344e78b1e4e48fa548ef8 from peer 0429a57df6ba4f92a73e779b04b05040 at {username='slave'} at 127.23.104.200:59587: session id = 0429a57df6ba4f92a73e779b04b05040-091475e28bb344e78b1e4e48fa548ef8
I20240503 02:13:37.457695 31604 tablet_copy_source_session.cc:205] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:37.468379 1461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 091475e28bb344e78b1e4e48fa548ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:37.484196 1461 tablet_copy_client.cc:783] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:37.484534 1461 tablet_copy_client.cc:647] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:37.486521 1477 tablet_copy_client.cc:1002] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:37.490085 31583 consensus_queue.cc:237] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:37.492551 31323 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.493314 31454 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:37.493817 31518 consensus_peers.cc:485] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 -> Peer 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997): Couldn't send request to peer 4a3e3937b61447598fd76bc57f5d0939. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17be048393974fdf9811c5b8f8a97579. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:37.493894 1314 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.494179 1336 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.497195 1314 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:37.497761 31323 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:37.505031 31048 catalog_manager.cc:5484] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:37.514660 1499 ts_tablet_manager.cc:902] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:37.521356 31454 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:37.529250 31323 consensus_queue.cc:237] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
W20240503 02:13:37.538197 31260 consensus_peers.cc:485] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401): Couldn't send request to peer 0429a57df6ba4f92a73e779b04b05040. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b76fa26b48e942ed840b5ee55ce40387. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:37.540534 31191 raft_consensus.cc:1261] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.541186 989 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.544281 1501 ts_tablet_manager.cc:902] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:37.544368 31583 raft_consensus.cc:1261] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.545010 1407 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.549574 1407 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:37.552109 31191 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:37.566164 31044 catalog_manager.cc:5484] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:37.582230 1499 tablet_copy_client.cc:308] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:37.583966 31582 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:37.605125 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet 9b909a66859144f7b4b649d505c9645e from peer 13d0f62249b34b01a3a47c6afe10e750 ({username='slave'} at 127.23.104.197:33983)
I20240503 02:13:37.605417 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet 9b909a66859144f7b4b649d505c9645e from peer 13d0f62249b34b01a3a47c6afe10e750 at {username='slave'} at 127.23.104.197:33983: session id = 13d0f62249b34b01a3a47c6afe10e750-9b909a66859144f7b4b649d505c9645e
I20240503 02:13:37.608297 31344 tablet_copy_source_session.cc:205] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:37.613938 1499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b909a66859144f7b4b649d505c9645e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:37.623476 1499 tablet_copy_client.cc:783] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:37.623986 1499 tablet_copy_client.cc:647] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:37.626468 1508 tablet_copy_client.cc:1002] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:37.644291 31454 consensus_queue.cc:237] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:37.648598 31582 raft_consensus.cc:1261] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.649303 1294 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.649680 31323 raft_consensus.cc:1261] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:37.650931 31388 consensus_peers.cc:485] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 -> Peer 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543): Couldn't send request to peer 13d0f62249b34b01a3a47c6afe10e750. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9df1587246a4df19fda13f4be2b1177. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:37.651191 853 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.668771 1396 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:37.669667 31323 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:37.673805 1520 ts_tablet_manager.cc:902] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630: Initiating tablet copy from peer fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:37.679986 1501 tablet_copy_client.cc:308] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:37.681417 31044 catalog_manager.cc:5484] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:37.689895 1520 tablet_copy_client.cc:308] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:39101
I20240503 02:13:37.690222 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet 4dc576d8e72647c08736ae9508f6d27b from peer 0429a57df6ba4f92a73e779b04b05040 ({username='slave'} at 127.23.104.200:44199)
I20240503 02:13:37.690403 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet 4dc576d8e72647c08736ae9508f6d27b from peer 0429a57df6ba4f92a73e779b04b05040 at {username='slave'} at 127.23.104.200:44199: session id = 0429a57df6ba4f92a73e779b04b05040-4dc576d8e72647c08736ae9508f6d27b
I20240503 02:13:37.690711 31474 tablet_copy_service.cc:140] P fbb7d16e67d94775b2881712419c0577: Received BeginTabletCopySession request for tablet 37ca6554fec24a07865da3eabef27152 from peer 05ef0a15decc468ea3a1af8828b0f630 ({username='slave'} at 127.23.104.198:53527)
I20240503 02:13:37.690881 31474 tablet_copy_service.cc:161] P fbb7d16e67d94775b2881712419c0577: Beginning new tablet copy session on tablet 37ca6554fec24a07865da3eabef27152 from peer 05ef0a15decc468ea3a1af8828b0f630 at {username='slave'} at 127.23.104.198:53527: session id = 05ef0a15decc468ea3a1af8828b0f630-37ca6554fec24a07865da3eabef27152
I20240503 02:13:37.693243 31344 tablet_copy_source_session.cc:205] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:37.693159 31582 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:37.693751 31474 tablet_copy_source_session.cc:205] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:37.694917 1501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc576d8e72647c08736ae9508f6d27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:37.695099 1520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ca6554fec24a07865da3eabef27152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:37.708122 1520 tablet_copy_client.cc:783] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:37.708467 1520 tablet_copy_client.cc:647] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:37.710115 1525 tablet_copy_client.cc:1002] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:37.711215 1501 tablet_copy_client.cc:783] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:37.711526 1501 tablet_copy_client.cc:647] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:37.713225 1516 tablet_copy_client.cc:1002] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:37.714177 1530 ts_tablet_manager.cc:902] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:37.730680 1530 tablet_copy_client.cc:308] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:37.738889 31323 consensus_queue.cc:237] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:37.757963 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet 3f8353133ccd4601a3761652546a33b5 from peer 4a3e3937b61447598fd76bc57f5d0939 ({username='slave'} at 127.23.104.199:34985)
I20240503 02:13:37.758131 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet 3f8353133ccd4601a3761652546a33b5 from peer 4a3e3937b61447598fd76bc57f5d0939 at {username='slave'} at 127.23.104.199:34985: session id = 4a3e3937b61447598fd76bc57f5d0939-3f8353133ccd4601a3761652546a33b5
I20240503 02:13:37.761229 31344 tablet_copy_source_session.cc:205] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:37.763211 1530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8353133ccd4601a3761652546a33b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:37.771536 1537 ts_tablet_manager.cc:902] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750: Initiating tablet copy from peer fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:37.771608 31582 raft_consensus.cc:1261] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.772006 1530 tablet_copy_client.cc:783] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:37.772356 1530 tablet_copy_client.cc:647] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:37.772267 989 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.774343 1534 tablet_copy_client.cc:1002] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Injecting latency into file download: 1500
W20240503 02:13:37.777089 31259 consensus_peers.cc:485] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195): Couldn't send request to peer 05ef0a15decc468ea3a1af8828b0f630. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e964b80f5414d9ea3a5aea07b99d1d4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:37.778334 31453 raft_consensus.cc:1261] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.778395 989 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:37.779338 1407 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.780104 1539 ts_tablet_manager.cc:902] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040: Initiating tablet copy from peer b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:37.780678 31583 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:37.789194 31048 catalog_manager.cc:5484] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:37.796707 31453 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:37.810631 1539 tablet_copy_client.cc:308] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.193:44019
I20240503 02:13:37.829640 31213 tablet_copy_service.cc:140] P b52c8be37a2241ff9dc555aa0cebd5e5: Received BeginTabletCopySession request for tablet 0ce9e7d46a6e475faaffb1ee26973db7 from peer 0429a57df6ba4f92a73e779b04b05040 ({username='slave'} at 127.23.104.200:44419)
I20240503 02:13:37.829912 31213 tablet_copy_service.cc:161] P b52c8be37a2241ff9dc555aa0cebd5e5: Beginning new tablet copy session on tablet 0ce9e7d46a6e475faaffb1ee26973db7 from peer 0429a57df6ba4f92a73e779b04b05040 at {username='slave'} at 127.23.104.200:44419: session id = 0429a57df6ba4f92a73e779b04b05040-0ce9e7d46a6e475faaffb1ee26973db7
I20240503 02:13:37.831668 1537 tablet_copy_client.cc:308] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:39101
I20240503 02:13:37.833657 31213 tablet_copy_source_session.cc:205] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:37.835568 1539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce9e7d46a6e475faaffb1ee26973db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:37.836462 31583 consensus_queue.cc:237] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:37.838234 31474 tablet_copy_service.cc:140] P fbb7d16e67d94775b2881712419c0577: Received BeginTabletCopySession request for tablet 9000ce76eb154434a88b52e357497767 from peer 13d0f62249b34b01a3a47c6afe10e750 ({username='slave'} at 127.23.104.197:57809)
I20240503 02:13:37.838379 31474 tablet_copy_service.cc:161] P fbb7d16e67d94775b2881712419c0577: Beginning new tablet copy session on tablet 9000ce76eb154434a88b52e357497767 from peer 13d0f62249b34b01a3a47c6afe10e750 at {username='slave'} at 127.23.104.197:57809: session id = 13d0f62249b34b01a3a47c6afe10e750-9000ce76eb154434a88b52e357497767
I20240503 02:13:37.841068 31474 tablet_copy_source_session.cc:205] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:37.842548 1537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9000ce76eb154434a88b52e357497767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:37.843915 1539 tablet_copy_client.cc:783] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:37.844151 1539 tablet_copy_client.cc:647] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:37.846156 1551 tablet_copy_client.cc:1002] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Injecting latency into file download: 1500
W20240503 02:13:37.847692 31520 consensus_peers.cc:485] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 -> Peer 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401): Couldn't send request to peer 0429a57df6ba4f92a73e779b04b05040. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00e1dad5efda4e9eae55a8f9d80f0f96. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:37.848074 31191 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.849017 1314 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.849982 1537 tablet_copy_client.cc:783] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:37.850250 1537 tablet_copy_client.cc:647] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:37.850688 31454 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:37.851037 1444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20240503 02:13:37.851284 1560 ts_tablet_manager.cc:902] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
W20240503 02:13:37.851416 1444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:13:37.851449 1217 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.851945 1557 tablet_copy_client.cc:1002] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:37.855203 1217 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:37.860913 1560 tablet_copy_client.cc:308] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:37.861604 31453 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:37.861604 31191 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
W20240503 02:13:37.863045 1444 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:37.866879 31048 catalog_manager.cc:5484] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:37.876133 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet b6c71c82cf034bd68aa725e086c56ffb from peer 05ef0a15decc468ea3a1af8828b0f630 ({username='slave'} at 127.23.104.198:42437)
I20240503 02:13:37.876289 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet b6c71c82cf034bd68aa725e086c56ffb from peer 05ef0a15decc468ea3a1af8828b0f630 at {username='slave'} at 127.23.104.198:42437: session id = 05ef0a15decc468ea3a1af8828b0f630-b6c71c82cf034bd68aa725e086c56ffb
I20240503 02:13:37.878939 31344 tablet_copy_source_session.cc:205] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:37.880703 1560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c71c82cf034bd68aa725e086c56ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:37.887419 1560 tablet_copy_client.cc:783] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:37.890544 1560 tablet_copy_client.cc:647] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:37.892630 1566 tablet_copy_client.cc:1002] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:37.896857 31323 consensus_queue.cc:237] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:37.899344 31191 raft_consensus.cc:1261] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:37.899839 31258 consensus_peers.cc:485] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997): Couldn't send request to peer 4a3e3937b61447598fd76bc57f5d0939. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c681bdcf3074c54b1d533cf1f7d2b75. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:37.900103 1407 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.900487 31453 raft_consensus.cc:1261] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.901528 1407 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.914752 1357 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:37.915370 31192 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:37.921171 31453 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:37.922194 31044 catalog_manager.cc:5484] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:37.949956 31192 consensus_queue.cc:237] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:37.953383 31583 raft_consensus.cc:1261] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.953840 31323 raft_consensus.cc:1261] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:37.954066 1573 consensus_queue.cc:1035] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:37.956207 1573 consensus_queue.cc:1035] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:37.964357 31127 consensus_peers.cc:485] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 -> Peer 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543): Couldn't send request to peer 13d0f62249b34b01a3a47c6afe10e750. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f52b9a3c14a04815800753bda9505cee. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:37.966087 1451 raft_consensus.cc:2932] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:37.966672 31323 raft_consensus.cc:2932] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:37.969151 31583 raft_consensus.cc:2932] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:37.976311 31048 catalog_manager.cc:5484] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New cstate: current_term: 1 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:37.986255 1596 ts_tablet_manager.cc:902] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:38.031275 1596 tablet_copy_client.cc:308] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:38.032132 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet b76fa26b48e942ed840b5ee55ce40387 from peer 0429a57df6ba4f92a73e779b04b05040 ({username='slave'} at 127.23.104.200:44199)
I20240503 02:13:38.032318 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet b76fa26b48e942ed840b5ee55ce40387 from peer 0429a57df6ba4f92a73e779b04b05040 at {username='slave'} at 127.23.104.200:44199: session id = 0429a57df6ba4f92a73e779b04b05040-b76fa26b48e942ed840b5ee55ce40387
I20240503 02:13:38.035390 31344 tablet_copy_source_session.cc:205] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:38.036609 1596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76fa26b48e942ed840b5ee55ce40387. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:38.038992 31323 consensus_queue.cc:237] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:38.054057 1596 tablet_copy_client.cc:783] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:38.054397 1596 tablet_copy_client.cc:647] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:38.056167 31453 raft_consensus.cc:1261] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:38.056155 31583 raft_consensus.cc:1261] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:38.056566 31259 consensus_peers.cc:485] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195): Couldn't send request to peer 05ef0a15decc468ea3a1af8828b0f630. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7626c06dc67448f0b65f67b401d62c88. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:38.056772 1357 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:38.057396 1357 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:38.061255 1608 tablet_copy_client.cc:1002] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:38.061894 1357 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:38.067183 31453 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:38.067552 31048 catalog_manager.cc:5484] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:38.072185 31583 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:38.086128 1619 ts_tablet_manager.cc:902] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939: Initiating tablet copy from peer b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:38.089226 1619 tablet_copy_client.cc:308] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:39849
I20240503 02:13:38.089967 31604 tablet_copy_service.cc:140] P b92ff49cec804b1f801c11b64e3259b3: Received BeginTabletCopySession request for tablet 17be048393974fdf9811c5b8f8a97579 from peer 4a3e3937b61447598fd76bc57f5d0939 ({username='slave'} at 127.23.104.199:40937)
I20240503 02:13:38.090142 31604 tablet_copy_service.cc:161] P b92ff49cec804b1f801c11b64e3259b3: Beginning new tablet copy session on tablet 17be048393974fdf9811c5b8f8a97579 from peer 4a3e3937b61447598fd76bc57f5d0939 at {username='slave'} at 127.23.104.199:40937: session id = 4a3e3937b61447598fd76bc57f5d0939-17be048393974fdf9811c5b8f8a97579
I20240503 02:13:38.093170 31604 tablet_copy_source_session.cc:205] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:38.095628 1619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17be048393974fdf9811c5b8f8a97579. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:38.101663 1619 tablet_copy_client.cc:783] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:38.101914 1619 tablet_copy_client.cc:647] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:38.103539 1624 tablet_copy_client.cc:1002] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:38.125494 1635 ts_tablet_manager.cc:902] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750: Initiating tablet copy from peer fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:38.142446 1635 tablet_copy_client.cc:308] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:39101
I20240503 02:13:38.143205 31474 tablet_copy_service.cc:140] P fbb7d16e67d94775b2881712419c0577: Received BeginTabletCopySession request for tablet d9df1587246a4df19fda13f4be2b1177 from peer 13d0f62249b34b01a3a47c6afe10e750 ({username='slave'} at 127.23.104.197:57809)
I20240503 02:13:38.143373 31474 tablet_copy_service.cc:161] P fbb7d16e67d94775b2881712419c0577: Beginning new tablet copy session on tablet d9df1587246a4df19fda13f4be2b1177 from peer 13d0f62249b34b01a3a47c6afe10e750 at {username='slave'} at 127.23.104.197:57809: session id = 13d0f62249b34b01a3a47c6afe10e750-d9df1587246a4df19fda13f4be2b1177
I20240503 02:13:38.146138 31474 tablet_copy_source_session.cc:205] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:38.147241 1635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9df1587246a4df19fda13f4be2b1177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:38.153406 1635 tablet_copy_client.cc:783] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:38.153628 1635 tablet_copy_client.cc:647] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:38.157644 1649 tablet_copy_client.cc:1002] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:38.227437 1048 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:38.228981 1675 ts_tablet_manager.cc:902] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:38.255517 784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:38.264854 1684 ts_tablet_manager.cc:902] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040: Initiating tablet copy from peer b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:38.283552 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:38.285959 652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:38.297915 31171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:38.304033 1675 tablet_copy_client.cc:308] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:38.308054 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet 9e964b80f5414d9ea3a5aea07b99d1d4 from peer 05ef0a15decc468ea3a1af8828b0f630 ({username='slave'} at 127.23.104.198:42437)
I20240503 02:13:38.308223 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet 9e964b80f5414d9ea3a5aea07b99d1d4 from peer 05ef0a15decc468ea3a1af8828b0f630 at {username='slave'} at 127.23.104.198:42437: session id = 05ef0a15decc468ea3a1af8828b0f630-9e964b80f5414d9ea3a5aea07b99d1d4
I20240503 02:13:38.310304 31344 tablet_copy_source_session.cc:205] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:38.311635 1675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e964b80f5414d9ea3a5aea07b99d1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:38.316282 31563 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:38.332058 31303 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:38.333884 1675 tablet_copy_client.cc:783] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:38.334159 1675 tablet_copy_client.cc:647] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:38.336161 1692 tablet_copy_client.cc:1002] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:38.348235 1684 tablet_copy_client.cc:308] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:39849
I20240503 02:13:38.348906 31604 tablet_copy_service.cc:140] P b92ff49cec804b1f801c11b64e3259b3: Received BeginTabletCopySession request for tablet 00e1dad5efda4e9eae55a8f9d80f0f96 from peer 0429a57df6ba4f92a73e779b04b05040 ({username='slave'} at 127.23.104.200:59587)
I20240503 02:13:38.349061 31604 tablet_copy_service.cc:161] P b92ff49cec804b1f801c11b64e3259b3: Beginning new tablet copy session on tablet 00e1dad5efda4e9eae55a8f9d80f0f96 from peer 0429a57df6ba4f92a73e779b04b05040 at {username='slave'} at 127.23.104.200:59587: session id = 0429a57df6ba4f92a73e779b04b05040-00e1dad5efda4e9eae55a8f9d80f0f96
I20240503 02:13:38.353567 31604 tablet_copy_source_session.cc:205] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:38.354956 1684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e1dad5efda4e9eae55a8f9d80f0f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:38.355814 31433 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:38.391986 1700 ts_tablet_manager.cc:902] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750: Initiating tablet copy from peer b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:38.392768 1684 tablet_copy_client.cc:783] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:38.393045 1684 tablet_copy_client.cc:647] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:38.399338 1696 tablet_copy_client.cc:1002] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:38.454252 1700 tablet_copy_client.cc:308] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.193:44019
I20240503 02:13:38.466459 31213 tablet_copy_service.cc:140] P b52c8be37a2241ff9dc555aa0cebd5e5: Received BeginTabletCopySession request for tablet f52b9a3c14a04815800753bda9505cee from peer 13d0f62249b34b01a3a47c6afe10e750 ({username='slave'} at 127.23.104.197:48913)
I20240503 02:13:38.466734 31213 tablet_copy_service.cc:161] P b52c8be37a2241ff9dc555aa0cebd5e5: Beginning new tablet copy session on tablet f52b9a3c14a04815800753bda9505cee from peer 13d0f62249b34b01a3a47c6afe10e750 at {username='slave'} at 127.23.104.197:48913: session id = 13d0f62249b34b01a3a47c6afe10e750-f52b9a3c14a04815800753bda9505cee
I20240503 02:13:38.469902 31213 tablet_copy_source_session.cc:205] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:38.471207 1700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f52b9a3c14a04815800753bda9505cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:38.476722 1700 tablet_copy_client.cc:783] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:38.476974 1700 tablet_copy_client.cc:647] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:38.478807 1707 tablet_copy_client.cc:1002] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:38.489064 1722 ts_tablet_manager.cc:902] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
Runtime error: ksck discovered errors
I20240503 02:13:38.511116 1722 tablet_copy_client.cc:308] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:38.511921 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet 8c681bdcf3074c54b1d533cf1f7d2b75 from peer 4a3e3937b61447598fd76bc57f5d0939 ({username='slave'} at 127.23.104.199:34985)
I20240503 02:13:38.512060 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet 8c681bdcf3074c54b1d533cf1f7d2b75 from peer 4a3e3937b61447598fd76bc57f5d0939 at {username='slave'} at 127.23.104.199:34985: session id = 4a3e3937b61447598fd76bc57f5d0939-8c681bdcf3074c54b1d533cf1f7d2b75
I20240503 02:13:38.514617 31344 tablet_copy_source_session.cc:205] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:38.516597 1722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c681bdcf3074c54b1d533cf1f7d2b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:38.522089 1722 tablet_copy_client.cc:783] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:38.522383 1722 tablet_copy_client.cc:647] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:38.524657 1730 tablet_copy_client.cc:1002] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:38.586164 1748 ts_tablet_manager.cc:902] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:38.608059 1748 tablet_copy_client.cc:308] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:38.609340 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet 7626c06dc67448f0b65f67b401d62c88 from peer 05ef0a15decc468ea3a1af8828b0f630 ({username='slave'} at 127.23.104.198:42437)
I20240503 02:13:38.609544 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet 7626c06dc67448f0b65f67b401d62c88 from peer 05ef0a15decc468ea3a1af8828b0f630 at {username='slave'} at 127.23.104.198:42437: session id = 05ef0a15decc468ea3a1af8828b0f630-7626c06dc67448f0b65f67b401d62c88
I20240503 02:13:38.612707 31344 tablet_copy_source_session.cc:205] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:38.614063 1748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7626c06dc67448f0b65f67b401d62c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:38.620600 1748 tablet_copy_client.cc:783] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:38.620885 1748 tablet_copy_client.cc:647] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:38.623461 1755 tablet_copy_client.cc:1002] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:38.654183 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.201:0
--local_ip_for_outbound_sockets=127.23.104.201
--webserver_interface=127.23.104.201
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43173
--builtin_ntp_servers=127.23.104.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:13:38.661068 1414 tablet_copy_client.cc:523] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:38.664644 1414 tablet_bootstrap.cc:492] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap starting.
I20240503 02:13:38.677656 1414 log.cc:824] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:38.717664 1414 tablet_bootstrap.cc:492] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:38.718225 1414 tablet_bootstrap.cc:492] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap complete.
I20240503 02:13:38.718703 1414 ts_tablet_manager.cc:1367] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939: Time spent bootstrapping tablet: real 0.054s user 0.046s sys 0.001s
I20240503 02:13:38.725373 1414 raft_consensus.cc:348] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:38.725951 1414 raft_consensus.cc:729] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a3e3937b61447598fd76bc57f5d0939, State: Initialized, Role: LEARNER
I20240503 02:13:38.726483 1435 tablet_copy_client.cc:523] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:38.726883 1414 consensus_queue.cc:260] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:38.731253 1435 tablet_bootstrap.cc:492] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap starting.
I20240503 02:13:38.744458 1435 log.cc:824] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:38.746410 1414 ts_tablet_manager.cc:1398] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939: Time spent starting tablet: real 0.028s user 0.021s sys 0.007s
I20240503 02:13:38.749007 31604 tablet_copy_service.cc:342] P b92ff49cec804b1f801c11b64e3259b3: Request end of tablet copy session 4a3e3937b61447598fd76bc57f5d0939-c185751912de4f78808785a620dd9746 received from {username='slave'} at 127.23.104.199:40937
I20240503 02:13:38.749187 31604 tablet_copy_service.cc:433] P b92ff49cec804b1f801c11b64e3259b3: ending tablet copy session 4a3e3937b61447598fd76bc57f5d0939-c185751912de4f78808785a620dd9746 on tablet c185751912de4f78808785a620dd9746 with peer 4a3e3937b61447598fd76bc57f5d0939
I20240503 02:13:38.793928 1435 tablet_bootstrap.cc:492] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:38.794430 1435 tablet_bootstrap.cc:492] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap complete.
I20240503 02:13:38.794829 1435 ts_tablet_manager.cc:1367] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750: Time spent bootstrapping tablet: real 0.064s user 0.045s sys 0.012s
I20240503 02:13:38.799517 1435 raft_consensus.cc:348] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:38.799912 1435 raft_consensus.cc:729] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d0f62249b34b01a3a47c6afe10e750, State: Initialized, Role: LEARNER
I20240503 02:13:38.800369 1435 consensus_queue.cc:260] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:38.801925 1435 ts_tablet_manager.cc:1398] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20240503 02:13:38.802767 31604 tablet_copy_service.cc:342] P b92ff49cec804b1f801c11b64e3259b3: Request end of tablet copy session 13d0f62249b34b01a3a47c6afe10e750-80a15c3541fc4068acac595bd909f8d8 received from {username='slave'} at 127.23.104.197:53893
I20240503 02:13:38.802923 31604 tablet_copy_service.cc:433] P b92ff49cec804b1f801c11b64e3259b3: ending tablet copy session 13d0f62249b34b01a3a47c6afe10e750-80a15c3541fc4068acac595bd909f8d8 on tablet 80a15c3541fc4068acac595bd909f8d8 with peer 13d0f62249b34b01a3a47c6afe10e750
I20240503 02:13:38.927827 1457 tablet_copy_client.cc:523] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:38.931924 1457 tablet_bootstrap.cc:492] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap starting.
I20240503 02:13:38.944250 1457 log.cc:824] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:38.987104 1461 tablet_copy_client.cc:523] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:38.989687 1457 tablet_bootstrap.cc:492] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:38.990350 1457 tablet_bootstrap.cc:492] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap complete.
I20240503 02:13:38.990878 1457 ts_tablet_manager.cc:1367] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630: Time spent bootstrapping tablet: real 0.059s user 0.052s sys 0.006s
I20240503 02:13:38.995100 1461 tablet_bootstrap.cc:492] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap starting.
I20240503 02:13:39.011994 1457 raft_consensus.cc:348] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:39.012611 1457 raft_consensus.cc:729] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05ef0a15decc468ea3a1af8828b0f630, State: Initialized, Role: LEARNER
I20240503 02:13:39.013209 1457 consensus_queue.cc:260] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:39.029111 1457 ts_tablet_manager.cc:1398] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630: Time spent starting tablet: real 0.038s user 0.010s sys 0.000s
I20240503 02:13:39.030161 31474 tablet_copy_service.cc:342] P fbb7d16e67d94775b2881712419c0577: Request end of tablet copy session 05ef0a15decc468ea3a1af8828b0f630-7230f1d648c34351aed29f98795150a2 received from {username='slave'} at 127.23.104.198:53527
I20240503 02:13:39.030306 31474 tablet_copy_service.cc:433] P fbb7d16e67d94775b2881712419c0577: ending tablet copy session 05ef0a15decc468ea3a1af8828b0f630-7230f1d648c34351aed29f98795150a2 on tablet 7230f1d648c34351aed29f98795150a2 with peer 05ef0a15decc468ea3a1af8828b0f630
I20240503 02:13:39.037041 1461 log.cc:824] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:39.043201 936 raft_consensus.cc:1203] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240503 02:13:39.088683 1768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:39.088968 1768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:39.089157 1768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:13:39.096813 1461 tablet_bootstrap.cc:492] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:39.099673 1461 tablet_bootstrap.cc:492] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap complete.
W20240503 02:13:39.099928 1768 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:39.100103 1768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20240503 02:13:39.100286 1461 ts_tablet_manager.cc:1367] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040: Time spent bootstrapping tablet: real 0.105s user 0.051s sys 0.012s
W20240503 02:13:39.100351 1768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.201
I20240503 02:13:39.106508 1461 raft_consensus.cc:348] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:39.107201 1461 raft_consensus.cc:729] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0429a57df6ba4f92a73e779b04b05040, State: Initialized, Role: LEARNER
I20240503 02:13:39.107863 1461 consensus_queue.cc:260] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:39.114395 1768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.201:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.23.104.201
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.23.104.254:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.201
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:39.116111 1768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:39.118491 1768 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:13:39.126607 1461 ts_tablet_manager.cc:1398] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040: Time spent starting tablet: real 0.026s user 0.011s sys 0.009s
I20240503 02:13:39.127053 1499 tablet_copy_client.cc:523] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:39.128295 31604 tablet_copy_service.cc:342] P b92ff49cec804b1f801c11b64e3259b3: Request end of tablet copy session 0429a57df6ba4f92a73e779b04b05040-091475e28bb344e78b1e4e48fa548ef8 received from {username='slave'} at 127.23.104.200:59587
I20240503 02:13:39.128438 31604 tablet_copy_service.cc:433] P b92ff49cec804b1f801c11b64e3259b3: ending tablet copy session 0429a57df6ba4f92a73e779b04b05040-091475e28bb344e78b1e4e48fa548ef8 on tablet 091475e28bb344e78b1e4e48fa548ef8 with peer 0429a57df6ba4f92a73e779b04b05040
I20240503 02:13:39.130296 1499 tablet_bootstrap.cc:492] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap starting.
W20240503 02:13:39.147858 1833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:39.149060 1835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:39.158682 1838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:39.160262 1768 server_base.cc:1034] running on GCE node
I20240503 02:13:39.161228 1768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:39.162381 1768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:39.169871 1768 hybrid_clock.cc:648] HybridClock initialized: now 1714702419170504 us; error 785 us; skew 500 ppm
I20240503 02:13:39.170305 1768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:39.176676 1499 tablet_bootstrap.cc:492] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:39.177310 1499 tablet_bootstrap.cc:492] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap complete.
I20240503 02:13:39.177737 1499 ts_tablet_manager.cc:1367] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750: Time spent bootstrapping tablet: real 0.047s user 0.033s sys 0.012s
I20240503 02:13:39.178759 1499 raft_consensus.cc:348] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:39.179138 1499 raft_consensus.cc:729] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d0f62249b34b01a3a47c6afe10e750, State: Initialized, Role: LEARNER
I20240503 02:13:39.179229 1768 webserver.cc:460] Webserver started at http://127.23.104.201:33837/ using document root <none> and password file <none>
I20240503 02:13:39.179421 1499 consensus_queue.cc:260] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:39.180415 1768 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:39.180593 1768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:39.180663 1499 ts_tablet_manager.cc:1398] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:39.181026 1768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:13:39.181526 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 13d0f62249b34b01a3a47c6afe10e750-9b909a66859144f7b4b649d505c9645e received from {username='slave'} at 127.23.104.197:33983
I20240503 02:13:39.181674 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 13d0f62249b34b01a3a47c6afe10e750-9b909a66859144f7b4b649d505c9645e on tablet 9b909a66859144f7b4b649d505c9645e with peer 13d0f62249b34b01a3a47c6afe10e750
I20240503 02:13:39.185046 1768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "6ad20578b4a74bfabb6d1404de034a65"
format_stamp: "Formatted at 2024-05-03 02:13:39 on dist-test-slave-38ht"
I20240503 02:13:39.185889 1768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "6ad20578b4a74bfabb6d1404de034a65"
format_stamp: "Formatted at 2024-05-03 02:13:39 on dist-test-slave-38ht"
I20240503 02:13:39.192350 1768 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240503 02:13:39.196050 1858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:39.197872 1768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:39.198065 1768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "6ad20578b4a74bfabb6d1404de034a65"
format_stamp: "Formatted at 2024-05-03 02:13:39 on dist-test-slave-38ht"
I20240503 02:13:39.198323 1768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:39.210615 1520 tablet_copy_client.cc:523] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:39.213611 1501 tablet_copy_client.cc:523] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:39.214112 1520 tablet_bootstrap.cc:492] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap starting.
I20240503 02:13:39.216634 1501 tablet_bootstrap.cc:492] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040: Bootstrap starting.
I20240503 02:13:39.253286 673 raft_consensus.cc:1203] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:39.253867 1768 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:13:39.254940 1768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:39.255332 1768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:39.259337 1068 raft_consensus.cc:1203] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:39.260222 804 raft_consensus.cc:1203] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:39.274821 1530 tablet_copy_client.cc:523] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:39.280725 1530 tablet_bootstrap.cc:492] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap starting.
I20240503 02:13:39.280884 1501 tablet_bootstrap.cc:492] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:39.281508 1501 tablet_bootstrap.cc:492] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040: Bootstrap complete.
I20240503 02:13:39.281900 1501 ts_tablet_manager.cc:1367] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040: Time spent bootstrapping tablet: real 0.065s user 0.034s sys 0.012s
I20240503 02:13:39.283146 1501 raft_consensus.cc:348] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:39.283524 1501 raft_consensus.cc:729] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0429a57df6ba4f92a73e779b04b05040, State: Initialized, Role: LEARNER
I20240503 02:13:39.283818 1501 consensus_queue.cc:260] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:39.284515 1501 ts_tablet_manager.cc:1398] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:39.286924 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 0429a57df6ba4f92a73e779b04b05040-4dc576d8e72647c08736ae9508f6d27b received from {username='slave'} at 127.23.104.200:44199
I20240503 02:13:39.287081 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 0429a57df6ba4f92a73e779b04b05040-4dc576d8e72647c08736ae9508f6d27b on tablet 4dc576d8e72647c08736ae9508f6d27b with peer 0429a57df6ba4f92a73e779b04b05040
I20240503 02:13:39.296253 1768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:39.308456 1520 tablet_bootstrap.cc:492] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:39.309473 1520 tablet_bootstrap.cc:492] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap complete.
I20240503 02:13:39.309473 1768 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:39.309579 1768 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:39.309741 1768 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:39.309865 1768 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:13:39.309903 1520 ts_tablet_manager.cc:1367] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630: Time spent bootstrapping tablet: real 0.096s user 0.053s sys 0.000s
I20240503 02:13:39.314015 1520 raft_consensus.cc:348] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:39.314468 1520 raft_consensus.cc:729] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05ef0a15decc468ea3a1af8828b0f630, State: Initialized, Role: LEARNER
I20240503 02:13:39.314857 1520 consensus_queue.cc:260] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:39.318651 1520 ts_tablet_manager.cc:1398] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:13:39.322204 31474 tablet_copy_service.cc:342] P fbb7d16e67d94775b2881712419c0577: Request end of tablet copy session 05ef0a15decc468ea3a1af8828b0f630-37ca6554fec24a07865da3eabef27152 received from {username='slave'} at 127.23.104.198:53527
I20240503 02:13:39.322356 31474 tablet_copy_service.cc:433] P fbb7d16e67d94775b2881712419c0577: ending tablet copy session 05ef0a15decc468ea3a1af8828b0f630-37ca6554fec24a07865da3eabef27152 on tablet 37ca6554fec24a07865da3eabef27152 with peer 05ef0a15decc468ea3a1af8828b0f630
I20240503 02:13:39.346547 1530 tablet_bootstrap.cc:492] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:39.346599 1539 tablet_copy_client.cc:523] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:39.347117 1530 tablet_bootstrap.cc:492] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap complete.
I20240503 02:13:39.347414 1530 ts_tablet_manager.cc:1367] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939: Time spent bootstrapping tablet: real 0.067s user 0.053s sys 0.007s
I20240503 02:13:39.348613 1530 raft_consensus.cc:348] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:39.348882 1530 raft_consensus.cc:729] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a3e3937b61447598fd76bc57f5d0939, State: Initialized, Role: LEARNER
I20240503 02:13:39.349104 1530 consensus_queue.cc:260] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:39.349781 1530 ts_tablet_manager.cc:1398] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:39.350685 1539 tablet_bootstrap.cc:492] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap starting.
I20240503 02:13:39.352340 1537 tablet_copy_client.cc:523] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:39.353152 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 4a3e3937b61447598fd76bc57f5d0939-3f8353133ccd4601a3761652546a33b5 received from {username='slave'} at 127.23.104.199:34985
I20240503 02:13:39.353255 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 4a3e3937b61447598fd76bc57f5d0939-3f8353133ccd4601a3761652546a33b5 on tablet 3f8353133ccd4601a3761652546a33b5 with peer 4a3e3937b61447598fd76bc57f5d0939
I20240503 02:13:39.360555 1537 tablet_bootstrap.cc:492] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap starting.
I20240503 02:13:39.393049 1560 tablet_copy_client.cc:523] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:39.397593 1560 tablet_bootstrap.cc:492] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap starting.
I20240503 02:13:39.406457 804 raft_consensus.cc:1203] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:39.480216 1539 tablet_bootstrap.cc:492] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:39.480921 1539 tablet_bootstrap.cc:492] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap complete.
I20240503 02:13:39.481334 1539 ts_tablet_manager.cc:1367] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040: Time spent bootstrapping tablet: real 0.131s user 0.086s sys 0.021s
I20240503 02:13:39.482378 1539 raft_consensus.cc:348] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:39.482882 1539 raft_consensus.cc:729] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0429a57df6ba4f92a73e779b04b05040, State: Initialized, Role: LEARNER
I20240503 02:13:39.483206 1539 consensus_queue.cc:260] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:39.484215 1539 ts_tablet_manager.cc:1398] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:39.485208 31213 tablet_copy_service.cc:342] P b52c8be37a2241ff9dc555aa0cebd5e5: Request end of tablet copy session 0429a57df6ba4f92a73e779b04b05040-0ce9e7d46a6e475faaffb1ee26973db7 received from {username='slave'} at 127.23.104.200:44419
I20240503 02:13:39.485423 31213 tablet_copy_service.cc:433] P b52c8be37a2241ff9dc555aa0cebd5e5: ending tablet copy session 0429a57df6ba4f92a73e779b04b05040-0ce9e7d46a6e475faaffb1ee26973db7 on tablet 0ce9e7d46a6e475faaffb1ee26973db7 with peer 0429a57df6ba4f92a73e779b04b05040
I20240503 02:13:39.514999 1560 tablet_bootstrap.cc:492] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:39.515525 1560 tablet_bootstrap.cc:492] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap complete.
I20240503 02:13:39.515797 1560 ts_tablet_manager.cc:1367] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630: Time spent bootstrapping tablet: real 0.118s user 0.048s sys 0.004s
I20240503 02:13:39.516605 1560 raft_consensus.cc:348] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:39.516948 1560 raft_consensus.cc:729] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05ef0a15decc468ea3a1af8828b0f630, State: Initialized, Role: LEARNER
I20240503 02:13:39.517189 1560 consensus_queue.cc:260] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:39.517777 1560 ts_tablet_manager.cc:1398] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:39.519073 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 05ef0a15decc468ea3a1af8828b0f630-b6c71c82cf034bd68aa725e086c56ffb received from {username='slave'} at 127.23.104.198:42437
I20240503 02:13:39.519191 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 05ef0a15decc468ea3a1af8828b0f630-b6c71c82cf034bd68aa725e086c56ffb on tablet b6c71c82cf034bd68aa725e086c56ffb with peer 05ef0a15decc468ea3a1af8828b0f630
I20240503 02:13:39.521457 1537 tablet_bootstrap.cc:492] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:39.521935 1537 tablet_bootstrap.cc:492] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap complete.
I20240503 02:13:39.522254 1537 ts_tablet_manager.cc:1367] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750: Time spent bootstrapping tablet: real 0.162s user 0.063s sys 0.000s
I20240503 02:13:39.523914 1537 raft_consensus.cc:348] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:39.524185 1537 raft_consensus.cc:729] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d0f62249b34b01a3a47c6afe10e750, State: Initialized, Role: LEARNER
I20240503 02:13:39.524556 1537 consensus_queue.cc:260] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:39.525142 1537 ts_tablet_manager.cc:1398] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:13:39.525820 31474 tablet_copy_service.cc:342] P fbb7d16e67d94775b2881712419c0577: Request end of tablet copy session 13d0f62249b34b01a3a47c6afe10e750-9000ce76eb154434a88b52e357497767 received from {username='slave'} at 127.23.104.197:57809
I20240503 02:13:39.525941 31474 tablet_copy_service.cc:433] P fbb7d16e67d94775b2881712419c0577: ending tablet copy session 13d0f62249b34b01a3a47c6afe10e750-9000ce76eb154434a88b52e357497767 on tablet 9000ce76eb154434a88b52e357497767 with peer 13d0f62249b34b01a3a47c6afe10e750
I20240503 02:13:39.535718 1768 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.201:40601
I20240503 02:13:39.535795 2016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.201:40601 every 8 connection(s)
I20240503 02:13:39.537211 1768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240503 02:13:39.542627 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 1768
I20240503 02:13:39.542814 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240503 02:13:39.550949 2017 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43173
I20240503 02:13:39.551236 2017 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:39.552201 2017 heartbeater.cc:511] Master 127.23.104.254:43173 requested a full tablet report, sending...
I20240503 02:13:39.553486 31048 ts_manager.cc:194] Registered new tserver with Master: 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601)
I20240503 02:13:39.554220 31048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.201:36015
I20240503 02:13:39.559870 1217 raft_consensus.cc:1050] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3: attempting to promote NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 to VOTER
I20240503 02:13:39.560763 1217 consensus_queue.cc:237] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:39.561610 1596 tablet_copy_client.cc:523] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:39.563122 31453 raft_consensus.cc:1261] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:39.563560 31323 raft_consensus.cc:1261] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:39.563630 1336 consensus_queue.cc:1035] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:39.564456 936 raft_consensus.cc:1261] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:39.565075 1596 tablet_bootstrap.cc:492] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap starting.
I20240503 02:13:39.565299 1336 consensus_queue.cc:1035] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:39.565754 1336 consensus_queue.cc:1035] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:39.571075 673 raft_consensus.cc:1203] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:39.580973 1314 raft_consensus.cc:2932] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:39.585865 31453 raft_consensus.cc:2932] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:39.586591 31048 catalog_manager.cc:5484] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:39.587863 936 raft_consensus.cc:2932] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:39.592944 31323 raft_consensus.cc:2932] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:39.603912 1619 tablet_copy_client.cc:523] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:39.607237 1619 tablet_bootstrap.cc:492] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap starting.
I20240503 02:13:39.609145 936 raft_consensus.cc:1203] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:39.615206 31583 consensus_queue.cc:237] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:39.615856 1596 tablet_bootstrap.cc:492] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:39.616328 1596 tablet_bootstrap.cc:492] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap complete.
I20240503 02:13:39.616653 1596 ts_tablet_manager.cc:1367] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040: Time spent bootstrapping tablet: real 0.052s user 0.035s sys 0.017s
I20240503 02:13:39.617056 936 raft_consensus.cc:1261] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:39.617523 1596 raft_consensus.cc:348] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:39.617754 31453 raft_consensus.cc:1261] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:39.617758 2022 consensus_queue.cc:1035] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:39.617861 1596 raft_consensus.cc:729] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0429a57df6ba4f92a73e779b04b05040, State: Initialized, Role: LEARNER
I20240503 02:13:39.618073 1596 consensus_queue.cc:260] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:39.618346 1217 consensus_queue.cc:1035] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:39.619777 1596 ts_tablet_manager.cc:1398] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:39.620798 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 0429a57df6ba4f92a73e779b04b05040-b76fa26b48e942ed840b5ee55ce40387 received from {username='slave'} at 127.23.104.200:44199
I20240503 02:13:39.620939 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 0429a57df6ba4f92a73e779b04b05040-b76fa26b48e942ed840b5ee55ce40387 on tablet b76fa26b48e942ed840b5ee55ce40387 with peer 0429a57df6ba4f92a73e779b04b05040
I20240503 02:13:39.621888 2022 raft_consensus.cc:2932] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:39.622550 31453 raft_consensus.cc:2932] T c185751912de4f78808785a620dd9746 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:39.625257 31039 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c185751912de4f78808785a620dd9746 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:39.625707 936 raft_consensus.cc:2932] T c185751912de4f78808785a620dd9746 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:39.628672 31048 catalog_manager.cc:5484] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:39.641558 1336 raft_consensus.cc:1050] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3: attempting to promote NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 to VOTER
I20240503 02:13:39.642356 1336 consensus_queue.cc:237] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:39.658069 1635 tablet_copy_client.cc:523] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:39.658528 31303 tablet_service.cc:1514] Processing DeleteTablet for tablet c185751912de4f78808785a620dd9746 with delete_type TABLET_DATA_TOMBSTONED (TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55596
I20240503 02:13:39.662400 1635 tablet_bootstrap.cc:492] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap starting.
I20240503 02:13:39.665196 2033 tablet_replica.cc:330] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: stopping tablet replica
I20240503 02:13:39.665727 2033 raft_consensus.cc:2229] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:39.666111 2033 raft_consensus.cc:2258] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:39.668614 673 raft_consensus.cc:1261] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:39.669525 1217 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:39.673646 31191 raft_consensus.cc:1261] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:39.674217 1314 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:39.675356 31323 raft_consensus.cc:1261] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:39.675900 1314 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:39.676967 2033 ts_tablet_manager.cc:1875] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:39.676879 1314 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:39.677882 31191 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:39.685393 31049 catalog_manager.cc:5484] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:39.685878 673 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:39.688269 31323 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:39.689270 1619 tablet_bootstrap.cc:492] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:39.689671 1619 tablet_bootstrap.cc:492] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap complete.
I20240503 02:13:39.689914 1619 ts_tablet_manager.cc:1367] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939: Time spent bootstrapping tablet: real 0.083s user 0.053s sys 0.000s
I20240503 02:13:39.690665 1619 raft_consensus.cc:348] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:39.693642 1619 raft_consensus.cc:729] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a3e3937b61447598fd76bc57f5d0939, State: Initialized, Role: LEARNER
I20240503 02:13:39.694085 1619 consensus_queue.cc:260] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:39.695094 1619 ts_tablet_manager.cc:1398] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:13:39.695300 2033 ts_tablet_manager.cc:1888] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:39.695530 2033 log.cc:1196] T c185751912de4f78808785a620dd9746 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/c185751912de4f78808785a620dd9746
I20240503 02:13:39.696599 31604 tablet_copy_service.cc:342] P b92ff49cec804b1f801c11b64e3259b3: Request end of tablet copy session 4a3e3937b61447598fd76bc57f5d0939-17be048393974fdf9811c5b8f8a97579 received from {username='slave'} at 127.23.104.199:40937
I20240503 02:13:39.696774 31604 tablet_copy_service.cc:433] P b92ff49cec804b1f801c11b64e3259b3: ending tablet copy session 4a3e3937b61447598fd76bc57f5d0939-17be048393974fdf9811c5b8f8a97579 on tablet 17be048393974fdf9811c5b8f8a97579 with peer 4a3e3937b61447598fd76bc57f5d0939
I20240503 02:13:39.697187 31583 consensus_queue.cc:237] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:39.698642 31037 catalog_manager.cc:4830] TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): tablet c185751912de4f78808785a620dd9746 (table rebalance_test_table_1 [id=d3a62fd4d46c4ba082abbec8f97c84e0]) successfully deleted
I20240503 02:13:39.699906 31323 raft_consensus.cc:1261] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:39.700872 2022 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:39.703002 673 raft_consensus.cc:1261] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:39.703733 1217 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:39.704133 1336 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:39.704375 31323 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:39.705834 673 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:39.708073 31039 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80a15c3541fc4068acac595bd909f8d8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:39.710124 31049 catalog_manager.cc:5484] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:39.715356 31171 tablet_service.cc:1514] Processing DeleteTablet for tablet 80a15c3541fc4068acac595bd909f8d8 with delete_type TABLET_DATA_TOMBSTONED (TS b52c8be37a2241ff9dc555aa0cebd5e5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47568
I20240503 02:13:39.737980 2022 raft_consensus.cc:1050] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3: attempting to promote NON_VOTER 0429a57df6ba4f92a73e779b04b05040 to VOTER
I20240503 02:13:39.738761 2022 consensus_queue.cc:237] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:39.742527 2039 tablet_replica.cc:330] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5: stopping tablet replica
I20240503 02:13:39.743265 2039 raft_consensus.cc:2229] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:39.743863 2039 raft_consensus.cc:2258] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:39.747902 673 raft_consensus.cc:1203] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:39.749449 2039 ts_tablet_manager.cc:1875] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:39.751618 1068 raft_consensus.cc:1261] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:39.751761 31191 raft_consensus.cc:1261] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:39.751780 31453 raft_consensus.cc:1261] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:39.752355 1217 consensus_queue.cc:1035] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:39.752893 1217 consensus_queue.cc:1035] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:39.753000 1294 raft_consensus.cc:1050] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577: attempting to promote NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 to VOTER
I20240503 02:13:39.754026 1294 consensus_queue.cc:237] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:39.756028 1314 consensus_queue.cc:1035] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:39.760191 31192 raft_consensus.cc:1261] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:39.761528 804 raft_consensus.cc:1261] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:39.763151 853 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:39.763864 853 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:39.767076 31583 raft_consensus.cc:1261] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:39.766731 1217 raft_consensus.cc:2932] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:39.768090 1294 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:39.771667 1068 raft_consensus.cc:2932] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:39.778069 31046 catalog_manager.cc:5484] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040 reported cstate change: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:39.782231 31191 raft_consensus.cc:2932] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:39.782856 2039 ts_tablet_manager.cc:1888] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:39.783054 2039 log.cc:1196] T 80a15c3541fc4068acac595bd909f8d8 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/80a15c3541fc4068acac595bd909f8d8
I20240503 02:13:39.785356 804 raft_consensus.cc:1203] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:39.790690 31582 consensus_queue.cc:237] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:39.790482 31453 raft_consensus.cc:2932] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:39.792568 31038 catalog_manager.cc:4830] TS b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): tablet 80a15c3541fc4068acac595bd909f8d8 (table rebalance_test_table_0 [id=cc3c62c772ec4610892f67e8c6d52379]) successfully deleted
I20240503 02:13:39.793419 1068 raft_consensus.cc:1261] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:39.794277 31192 raft_consensus.cc:1261] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:39.794942 1217 consensus_queue.cc:1035] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:39.795333 2022 consensus_queue.cc:1035] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:39.797855 1217 raft_consensus.cc:2932] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:39.801426 1068 raft_consensus.cc:1203] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:39.803437 31039 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 091475e28bb344e78b1e4e48fa548ef8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:39.804766 31048 catalog_manager.cc:5484] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:39.806557 1068 raft_consensus.cc:2932] T 091475e28bb344e78b1e4e48fa548ef8 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
W20240503 02:13:39.809185 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:38195, user_credentials={real_user=slave}} blocked reactor thread for 46504.7us
I20240503 02:13:39.813058 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet 091475e28bb344e78b1e4e48fa548ef8 with delete_type TABLET_DATA_TOMBSTONED (TS fbb7d16e67d94775b2881712419c0577 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41348
I20240503 02:13:39.813724 31191 raft_consensus.cc:2932] T 091475e28bb344e78b1e4e48fa548ef8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:39.825740 2051 tablet_replica.cc:330] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577: stopping tablet replica
I20240503 02:13:39.826377 2051 raft_consensus.cc:2229] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:39.826187 1294 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:39.826920 2051 raft_consensus.cc:2258] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:39.831785 31582 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:39.832919 804 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:39.836562 1675 tablet_copy_client.cc:523] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:39.836392 31192 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:39.839995 1675 tablet_bootstrap.cc:492] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap starting.
I20240503 02:13:39.841102 2051 ts_tablet_manager.cc:1875] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:39.841260 31049 catalog_manager.cc:5484] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:39.852494 31453 consensus_queue.cc:237] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:39.854429 31582 raft_consensus.cc:1261] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:39.855070 803 raft_consensus.cc:1261] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:39.855049 2047 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:39.855634 2046 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:39.858400 1294 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:39.859103 31583 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:39.860819 2051 ts_tablet_manager.cc:1888] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:39.860944 2051 log.cc:1196] T 091475e28bb344e78b1e4e48fa548ef8 P fbb7d16e67d94775b2881712419c0577: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/091475e28bb344e78b1e4e48fa548ef8
I20240503 02:13:39.862974 1068 raft_consensus.cc:1203] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:39.865985 31036 catalog_manager.cc:4830] TS fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101): tablet 091475e28bb344e78b1e4e48fa548ef8 (table rebalance_test_table_3 [id=e0a6e74497494a20be21d627357e4eb0]) successfully deleted
I20240503 02:13:39.866247 31036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7230f1d648c34351aed29f98795150a2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:39.866354 31048 catalog_manager.cc:5484] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:39.870607 31171 tablet_service.cc:1514] Processing DeleteTablet for tablet 7230f1d648c34351aed29f98795150a2 with delete_type TABLET_DATA_TOMBSTONED (TS b52c8be37a2241ff9dc555aa0cebd5e5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47568
I20240503 02:13:39.870911 2039 tablet_replica.cc:330] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5: stopping tablet replica
I20240503 02:13:39.871080 2039 raft_consensus.cc:2229] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:39.871367 2039 raft_consensus.cc:2258] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:39.871815 804 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:39.873013 2039 ts_tablet_manager.cc:1875] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:39.882009 2039 ts_tablet_manager.cc:1888] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:39.882185 2039 log.cc:1196] T 7230f1d648c34351aed29f98795150a2 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/7230f1d648c34351aed29f98795150a2
I20240503 02:13:39.882874 1635 tablet_bootstrap.cc:492] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:39.883086 31038 catalog_manager.cc:4830] TS b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): tablet 7230f1d648c34351aed29f98795150a2 (table rebalance_test_table_8 [id=dc947496e186485092d5d621503a4e6f]) successfully deleted
I20240503 02:13:39.883236 1635 tablet_bootstrap.cc:492] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap complete.
I20240503 02:13:39.883466 1635 ts_tablet_manager.cc:1367] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750: Time spent bootstrapping tablet: real 0.221s user 0.069s sys 0.007s
I20240503 02:13:39.884281 1635 raft_consensus.cc:348] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:39.884506 1635 raft_consensus.cc:729] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d0f62249b34b01a3a47c6afe10e750, State: Initialized, Role: LEARNER
I20240503 02:13:39.884722 1635 consensus_queue.cc:260] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:39.885226 1635 ts_tablet_manager.cc:1398] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:39.886124 31474 tablet_copy_service.cc:342] P fbb7d16e67d94775b2881712419c0577: Request end of tablet copy session 13d0f62249b34b01a3a47c6afe10e750-d9df1587246a4df19fda13f4be2b1177 received from {username='slave'} at 127.23.104.197:57809
I20240503 02:13:39.886251 31474 tablet_copy_service.cc:433] P fbb7d16e67d94775b2881712419c0577: ending tablet copy session 13d0f62249b34b01a3a47c6afe10e750-d9df1587246a4df19fda13f4be2b1177 on tablet d9df1587246a4df19fda13f4be2b1177 with peer 13d0f62249b34b01a3a47c6afe10e750
I20240503 02:13:39.899741 1684 tablet_copy_client.cc:523] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:39.907987 1684 tablet_bootstrap.cc:492] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap starting.
I20240503 02:13:39.920647 1068 raft_consensus.cc:1203] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:39.933589 1675 tablet_bootstrap.cc:492] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:39.934206 1675 tablet_bootstrap.cc:492] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap complete.
I20240503 02:13:39.934610 1675 ts_tablet_manager.cc:1367] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630: Time spent bootstrapping tablet: real 0.095s user 0.045s sys 0.001s
I20240503 02:13:39.935683 1675 raft_consensus.cc:348] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:39.936041 1675 raft_consensus.cc:729] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05ef0a15decc468ea3a1af8828b0f630, State: Initialized, Role: LEARNER
I20240503 02:13:39.936333 1675 consensus_queue.cc:260] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:39.939047 1675 ts_tablet_manager.cc:1398] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:39.940071 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 05ef0a15decc468ea3a1af8828b0f630-9e964b80f5414d9ea3a5aea07b99d1d4 received from {username='slave'} at 127.23.104.198:42437
I20240503 02:13:39.940196 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 05ef0a15decc468ea3a1af8828b0f630-9e964b80f5414d9ea3a5aea07b99d1d4 on tablet 9e964b80f5414d9ea3a5aea07b99d1d4 with peer 05ef0a15decc468ea3a1af8828b0f630
I20240503 02:13:39.948920 853 raft_consensus.cc:1050] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577: attempting to promote NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 to VOTER
I20240503 02:13:39.949717 804 raft_consensus.cc:1203] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:39.950006 853 consensus_queue.cc:237] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:39.960987 31324 raft_consensus.cc:1261] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:39.961664 2047 consensus_queue.cc:1035] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:39.970391 804 raft_consensus.cc:1261] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:39.970976 2047 consensus_queue.cc:1035] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:39.979526 31192 raft_consensus.cc:1261] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:39.980264 2047 consensus_queue.cc:1035] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:39.982673 1700 tablet_copy_client.cc:523] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:39.986083 1700 tablet_bootstrap.cc:492] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap starting.
I20240503 02:13:40.000172 1684 tablet_bootstrap.cc:492] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:40.000604 1684 tablet_bootstrap.cc:492] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap complete.
I20240503 02:13:40.000896 1684 ts_tablet_manager.cc:1367] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040: Time spent bootstrapping tablet: real 0.093s user 0.056s sys 0.000s
I20240503 02:13:40.001756 1684 raft_consensus.cc:348] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:40.002045 1684 raft_consensus.cc:729] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0429a57df6ba4f92a73e779b04b05040, State: Initialized, Role: LEARNER
I20240503 02:13:40.002271 1684 consensus_queue.cc:260] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:40.002836 1684 ts_tablet_manager.cc:1398] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:40.004073 31604 tablet_copy_service.cc:342] P b92ff49cec804b1f801c11b64e3259b3: Request end of tablet copy session 0429a57df6ba4f92a73e779b04b05040-00e1dad5efda4e9eae55a8f9d80f0f96 received from {username='slave'} at 127.23.104.200:59587
I20240503 02:13:40.004251 31604 tablet_copy_service.cc:433] P b92ff49cec804b1f801c11b64e3259b3: ending tablet copy session 0429a57df6ba4f92a73e779b04b05040-00e1dad5efda4e9eae55a8f9d80f0f96 on tablet 00e1dad5efda4e9eae55a8f9d80f0f96 with peer 0429a57df6ba4f92a73e779b04b05040
I20240503 02:13:40.015643 2047 raft_consensus.cc:2932] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.018072 31191 raft_consensus.cc:2932] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.021713 31324 raft_consensus.cc:2932] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.023006 31048 catalog_manager.cc:5484] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:40.024999 1722 tablet_copy_client.cc:523] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:40.028153 31453 consensus_queue.cc:237] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:40.028621 1722 tablet_bootstrap.cc:492] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap starting.
I20240503 02:13:40.029867 803 raft_consensus.cc:1261] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.029970 31324 raft_consensus.cc:1261] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.030508 2047 consensus_queue.cc:1035] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.030674 853 consensus_queue.cc:1035] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.030757 804 raft_consensus.cc:2932] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.036131 1294 raft_consensus.cc:2932] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.037478 803 raft_consensus.cc:2932] T 37ca6554fec24a07865da3eabef27152 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.039276 31036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37ca6554fec24a07865da3eabef27152 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:40.039458 31324 raft_consensus.cc:2932] T 37ca6554fec24a07865da3eabef27152 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.040843 31046 catalog_manager.cc:5484] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:40.051229 31171 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ca6554fec24a07865da3eabef27152 with delete_type TABLET_DATA_TOMBSTONED (TS b52c8be37a2241ff9dc555aa0cebd5e5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47568
I20240503 02:13:40.051687 2039 tablet_replica.cc:330] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5: stopping tablet replica
I20240503 02:13:40.053507 2039 raft_consensus.cc:2229] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:40.053746 2039 raft_consensus.cc:2258] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:40.054723 2039 ts_tablet_manager.cc:1875] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:40.061302 2039 ts_tablet_manager.cc:1888] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:40.061412 2039 log.cc:1196] T 37ca6554fec24a07865da3eabef27152 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/37ca6554fec24a07865da3eabef27152
I20240503 02:13:40.062076 31038 catalog_manager.cc:4830] TS b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): tablet 37ca6554fec24a07865da3eabef27152 (table rebalance_test_table_0 [id=cc3c62c772ec4610892f67e8c6d52379]) successfully deleted
I20240503 02:13:40.065519 673 raft_consensus.cc:1203] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:40.091609 1722 tablet_bootstrap.cc:492] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:40.092078 1722 tablet_bootstrap.cc:492] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap complete.
I20240503 02:13:40.092343 1722 ts_tablet_manager.cc:1367] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939: Time spent bootstrapping tablet: real 0.064s user 0.044s sys 0.005s
I20240503 02:13:40.092669 1700 tablet_bootstrap.cc:492] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:40.093312 1700 tablet_bootstrap.cc:492] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap complete.
I20240503 02:13:40.093236 1722 raft_consensus.cc:348] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:40.093502 1722 raft_consensus.cc:729] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a3e3937b61447598fd76bc57f5d0939, State: Initialized, Role: LEARNER
I20240503 02:13:40.093642 1700 ts_tablet_manager.cc:1367] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750: Time spent bootstrapping tablet: real 0.108s user 0.044s sys 0.007s
I20240503 02:13:40.093739 1722 consensus_queue.cc:260] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:40.094449 1700 raft_consensus.cc:348] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:40.094702 1700 raft_consensus.cc:729] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d0f62249b34b01a3a47c6afe10e750, State: Initialized, Role: LEARNER
I20240503 02:13:40.094902 1700 consensus_queue.cc:260] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:40.095057 1722 ts_tablet_manager.cc:1398] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:13:40.095554 1700 ts_tablet_manager.cc:1398] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:40.095712 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 4a3e3937b61447598fd76bc57f5d0939-8c681bdcf3074c54b1d533cf1f7d2b75 received from {username='slave'} at 127.23.104.199:34985
I20240503 02:13:40.095882 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 4a3e3937b61447598fd76bc57f5d0939-8c681bdcf3074c54b1d533cf1f7d2b75 on tablet 8c681bdcf3074c54b1d533cf1f7d2b75 with peer 4a3e3937b61447598fd76bc57f5d0939
I20240503 02:13:40.100361 31213 tablet_copy_service.cc:342] P b52c8be37a2241ff9dc555aa0cebd5e5: Request end of tablet copy session 13d0f62249b34b01a3a47c6afe10e750-f52b9a3c14a04815800753bda9505cee received from {username='slave'} at 127.23.104.197:48913
I20240503 02:13:40.100502 31213 tablet_copy_service.cc:433] P b52c8be37a2241ff9dc555aa0cebd5e5: ending tablet copy session 13d0f62249b34b01a3a47c6afe10e750-f52b9a3c14a04815800753bda9505cee on tablet f52b9a3c14a04815800753bda9505cee with peer 13d0f62249b34b01a3a47c6afe10e750
I20240503 02:13:40.117223 1357 raft_consensus.cc:1050] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 to VOTER
I20240503 02:13:40.118232 1357 consensus_queue.cc:237] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:40.121096 31192 raft_consensus.cc:1261] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.121220 31582 raft_consensus.cc:1261] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.121317 673 raft_consensus.cc:1261] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.121659 989 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.122042 2082 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.122628 2090 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.124485 1748 tablet_copy_client.cc:523] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:40.127282 1748 tablet_bootstrap.cc:492] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap starting.
I20240503 02:13:40.129487 2090 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.130153 31583 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.132781 673 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.133836 31192 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.136781 31048 catalog_manager.cc:5484] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:40.142750 31324 consensus_queue.cc:237] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:40.145192 31192 raft_consensus.cc:1261] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.145715 989 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.145030 673 raft_consensus.cc:1261] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.147393 989 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.149708 989 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.150521 672 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.155911 31049 catalog_manager.cc:5484] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 reported cstate change: config changed from index 5 to 6, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.156855 31192 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.162207 31563 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b909a66859144f7b4b649d505c9645e with delete_type TABLET_DATA_TOMBSTONED (TS b92ff49cec804b1f801c11b64e3259b3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36462
I20240503 02:13:40.164145 936 raft_consensus.cc:1203] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:40.164316 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b909a66859144f7b4b649d505c9645e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:40.173085 2100 tablet_replica.cc:330] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3: stopping tablet replica
I20240503 02:13:40.173588 2100 raft_consensus.cc:2229] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:40.179250 2100 raft_consensus.cc:2258] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:40.181425 2100 ts_tablet_manager.cc:1875] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:40.188673 2090 raft_consensus.cc:1050] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 to VOTER
I20240503 02:13:40.189380 2090 consensus_queue.cc:237] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:40.191264 31454 raft_consensus.cc:1261] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.191533 31192 raft_consensus.cc:1261] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.191840 2082 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.192261 1357 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.194154 936 raft_consensus.cc:1261] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.194684 2090 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.195112 2100 ts_tablet_manager.cc:1888] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:40.196305 2082 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.199417 31454 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.202982 2100 log.cc:1196] T 9b909a66859144f7b4b649d505c9645e P b92ff49cec804b1f801c11b64e3259b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/9b909a66859144f7b4b649d505c9645e
I20240503 02:13:40.203106 31192 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.204856 31049 catalog_manager.cc:5484] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:40.208312 31039 catalog_manager.cc:4830] TS b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849): tablet 9b909a66859144f7b4b649d505c9645e (table rebalance_test_table_6 [id=ad6bdc3b37e64f27824a7eacb47ca790]) successfully deleted
I20240503 02:13:40.210703 1294 raft_consensus.cc:1050] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577: attempting to promote NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 to VOTER
I20240503 02:13:40.211511 1294 consensus_queue.cc:237] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:40.218252 673 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.218720 31583 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.219352 31324 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.219309 853 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.219820 853 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.220161 1294 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.225107 31323 consensus_queue.cc:237] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:40.226732 1748 tablet_bootstrap.cc:492] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:40.226850 935 raft_consensus.cc:1261] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.227193 1748 tablet_bootstrap.cc:492] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap complete.
I20240503 02:13:40.227389 31192 raft_consensus.cc:1261] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.227468 1748 ts_tablet_manager.cc:1367] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630: Time spent bootstrapping tablet: real 0.100s user 0.046s sys 0.007s
I20240503 02:13:40.228268 2090 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.229190 1748 raft_consensus.cc:348] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:40.229614 1748 raft_consensus.cc:729] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05ef0a15decc468ea3a1af8828b0f630, State: Initialized, Role: LEARNER
I20240503 02:13:40.229884 1748 consensus_queue.cc:260] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:40.230372 2090 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.231197 1748 ts_tablet_manager.cc:1398] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 02:13:40.231897 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 05ef0a15decc468ea3a1af8828b0f630-7626c06dc67448f0b65f67b401d62c88 received from {username='slave'} at 127.23.104.198:42437
I20240503 02:13:40.232007 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 05ef0a15decc468ea3a1af8828b0f630-7626c06dc67448f0b65f67b401d62c88 on tablet 7626c06dc67448f0b65f67b401d62c88 with peer 05ef0a15decc468ea3a1af8828b0f630
I20240503 02:13:40.231679 935 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.235919 2090 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.241243 31192 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.243800 2047 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.244488 31583 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.249085 31046 catalog_manager.cc:5484] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.249536 673 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.252166 989 raft_consensus.cc:1050] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 0429a57df6ba4f92a73e779b04b05040 to VOTER
I20240503 02:13:40.252983 989 consensus_queue.cc:237] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:40.258057 31046 catalog_manager.cc:5484] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.259625 31454 consensus_queue.cc:237] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:40.262949 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3f8353133ccd4601a3761652546a33b5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:40.264992 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f8353133ccd4601a3761652546a33b5 with delete_type TABLET_DATA_TOMBSTONED (TS fbb7d16e67d94775b2881712419c0577 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41348
I20240503 02:13:40.264680 935 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.265861 2051 tablet_replica.cc:330] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577: stopping tablet replica
I20240503 02:13:40.265924 31583 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.266110 2051 raft_consensus.cc:2229] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:40.266348 2051 raft_consensus.cc:2258] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:40.266813 673 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.266403 2047 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 02:13:40.267899 2051 ts_tablet_manager.cc:1875] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:40.270231 853 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.274294 2046 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.275472 1068 raft_consensus.cc:1261] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.276060 1357 consensus_queue.cc:1035] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.278043 2051 ts_tablet_manager.cc:1888] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:40.278221 2051 log.cc:1196] T 3f8353133ccd4601a3761652546a33b5 P fbb7d16e67d94775b2881712419c0577: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/3f8353133ccd4601a3761652546a33b5
I20240503 02:13:40.279100 31036 catalog_manager.cc:4830] TS fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101): tablet 3f8353133ccd4601a3761652546a33b5 (table rebalance_test_table_8 [id=dc947496e186485092d5d621503a4e6f]) successfully deleted
I20240503 02:13:40.280011 673 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.280718 31192 raft_consensus.cc:1261] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.281253 1357 consensus_queue.cc:1035] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.281531 31036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9000ce76eb154434a88b52e357497767 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:40.282788 31583 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.285161 31044 catalog_manager.cc:5484] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:40.290347 31303 tablet_service.cc:1514] Processing DeleteTablet for tablet 9000ce76eb154434a88b52e357497767 with delete_type TABLET_DATA_TOMBSTONED (TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55596
I20240503 02:13:40.291394 31583 raft_consensus.cc:1261] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.292029 989 consensus_queue.cc:1035] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.294013 989 raft_consensus.cc:2932] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.294411 2127 tablet_replica.cc:330] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: stopping tablet replica
I20240503 02:13:40.294601 2127 raft_consensus.cc:2229] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:40.294760 2127 pending_rounds.cc:62] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Trying to abort 1 pending ops.
I20240503 02:13:40.294871 2127 pending_rounds.cc:68] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023421113184280576 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "9000ce76eb154434a88b52e357497767" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } } }
I20240503 02:13:40.295245 2127 raft_consensus.cc:2894] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 02:13:40.295353 2127 raft_consensus.cc:2866] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:13:40.295485 2127 raft_consensus.cc:2258] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:40.296383 2127 ts_tablet_manager.cc:1875] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:40.296453 1068 raft_consensus.cc:2932] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.299060 31192 raft_consensus.cc:2932] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.302100 31046 catalog_manager.cc:5484] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 reported cstate change: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.304466 2127 ts_tablet_manager.cc:1888] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:40.304646 2127 log.cc:1196] T 9000ce76eb154434a88b52e357497767 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/9000ce76eb154434a88b52e357497767
I20240503 02:13:40.306521 31037 catalog_manager.cc:4830] TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): tablet 9000ce76eb154434a88b52e357497767 (table rebalance_test_table_5 [id=f64903d295f2485983d7a34ee454f2b9]) successfully deleted
I20240503 02:13:40.307534 31583 raft_consensus.cc:2932] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.329659 1068 raft_consensus.cc:1203] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:40.355284 936 raft_consensus.cc:1203] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:40.360617 1357 raft_consensus.cc:1050] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 to VOTER
I20240503 02:13:40.361330 1357 consensus_queue.cc:237] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:40.363385 31454 raft_consensus.cc:1261] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.363622 31583 raft_consensus.cc:1261] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.363629 803 raft_consensus.cc:1261] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.363915 989 consensus_queue.cc:1035] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.364205 2082 consensus_queue.cc:1035] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.364616 2090 consensus_queue.cc:1035] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.366787 804 raft_consensus.cc:1203] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:40.371635 1357 raft_consensus.cc:1050] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 0429a57df6ba4f92a73e779b04b05040 to VOTER
I20240503 02:13:40.372437 1357 consensus_queue.cc:237] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:40.374652 1068 raft_consensus.cc:1261] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.374852 31583 raft_consensus.cc:1261] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.375435 1357 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.376149 31192 raft_consensus.cc:1261] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.376472 2090 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.376919 2090 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.390230 1357 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.391886 989 raft_consensus.cc:2932] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.393996 1068 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.393988 31191 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.398358 31453 raft_consensus.cc:2932] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.404477 31048 catalog_manager.cc:5484] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 reported cstate change: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.404804 31044 catalog_manager.cc:5484] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.408946 803 raft_consensus.cc:2932] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.411432 31582 raft_consensus.cc:2932] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.419431 31324 consensus_queue.cc:237] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:40.421564 1783 raft_consensus.cc:1050] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5: attempting to promote NON_VOTER 0429a57df6ba4f92a73e779b04b05040 to VOTER
I20240503 02:13:40.422048 31582 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.422554 1783 consensus_queue.cc:237] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:40.423548 803 raft_consensus.cc:1261] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.424603 31583 raft_consensus.cc:1261] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.425066 2175 consensus_queue.cc:1035] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.425812 2175 consensus_queue.cc:1035] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.429657 1068 raft_consensus.cc:1261] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.429257 31324 consensus_queue.cc:237] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:40.430296 2072 consensus_queue.cc:1035] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.431102 31454 raft_consensus.cc:1261] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.431779 2082 raft_consensus.cc:2932] T b6c71c82cf034bd68aa725e086c56ffb P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.432255 2072 consensus_queue.cc:1035] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.432385 31582 raft_consensus.cc:1261] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.432998 1357 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.436559 803 raft_consensus.cc:2932] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.440378 2090 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.445523 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b6c71c82cf034bd68aa725e086c56ffb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:40.446419 31046 catalog_manager.cc:5484] T b6c71c82cf034bd68aa725e086c56ffb P 05ef0a15decc468ea3a1af8828b0f630 reported cstate change: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.451717 31582 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.452742 31583 raft_consensus.cc:2932] T b6c71c82cf034bd68aa725e086c56ffb P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.456197 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet b6c71c82cf034bd68aa725e086c56ffb with delete_type TABLET_DATA_TOMBSTONED (TS fbb7d16e67d94775b2881712419c0577 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41348
I20240503 02:13:40.456405 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b76fa26b48e942ed840b5ee55ce40387 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:40.456549 2051 tablet_replica.cc:330] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577: stopping tablet replica
I20240503 02:13:40.456754 2051 raft_consensus.cc:2229] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:40.457106 2051 raft_consensus.cc:2258] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:40.458725 1068 raft_consensus.cc:1261] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.459223 2051 ts_tablet_manager.cc:1875] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:40.459933 31048 catalog_manager.cc:5484] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.462527 2175 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.464041 31171 tablet_service.cc:1514] Processing DeleteTablet for tablet b76fa26b48e942ed840b5ee55ce40387 with delete_type TABLET_DATA_TOMBSTONED (TS b52c8be37a2241ff9dc555aa0cebd5e5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47568
I20240503 02:13:40.464359 2039 tablet_replica.cc:330] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5: stopping tablet replica
I20240503 02:13:40.464566 2039 raft_consensus.cc:2229] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:40.464851 2039 raft_consensus.cc:2258] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:40.466307 1068 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.466801 2039 ts_tablet_manager.cc:1875] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:40.466986 31583 raft_consensus.cc:1261] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.467545 1783 consensus_queue.cc:1035] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.468014 2051 ts_tablet_manager.cc:1888] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:40.468129 2051 log.cc:1196] T b6c71c82cf034bd68aa725e086c56ffb P fbb7d16e67d94775b2881712419c0577: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/b6c71c82cf034bd68aa725e086c56ffb
I20240503 02:13:40.471101 31036 catalog_manager.cc:4830] TS fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101): tablet b6c71c82cf034bd68aa725e086c56ffb (table rebalance_test_table_9 [id=3a5935f3f7924b5083fcfc4ee7015893]) successfully deleted
I20240503 02:13:40.472712 1357 raft_consensus.cc:1050] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 to VOTER
I20240503 02:13:40.473424 1357 consensus_queue.cc:237] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:40.477195 803 raft_consensus.cc:1261] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.477303 31453 raft_consensus.cc:1261] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.477969 989 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.478606 989 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.483644 31582 raft_consensus.cc:1261] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.484324 2175 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.486291 2039 ts_tablet_manager.cc:1888] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:40.486582 2039 log.cc:1196] T b76fa26b48e942ed840b5ee55ce40387 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/b76fa26b48e942ed840b5ee55ce40387
I20240503 02:13:40.487510 31038 catalog_manager.cc:4830] TS b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): tablet b76fa26b48e942ed840b5ee55ce40387 (table rebalance_test_table_2 [id=88d4dd05cfa84299b1e62ec2a8b6a517]) successfully deleted
I20240503 02:13:40.492969 2072 raft_consensus.cc:2932] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.496829 31583 raft_consensus.cc:2932] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.497520 2175 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.498536 31452 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.500526 31584 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.501915 1068 raft_consensus.cc:2932] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.504060 31049 catalog_manager.cc:5484] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.506108 31454 raft_consensus.cc:2932] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.507962 1294 raft_consensus.cc:1050] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577: attempting to promote NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 to VOTER
I20240503 02:13:40.508978 1294 consensus_queue.cc:237] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:40.514775 31192 consensus_queue.cc:237] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:40.515096 31044 catalog_manager.cc:5484] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.516283 803 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.518332 1068 raft_consensus.cc:1261] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.519479 31452 raft_consensus.cc:1261] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.519773 1783 consensus_queue.cc:1035] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.520339 2072 consensus_queue.cc:1035] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.523147 673 raft_consensus.cc:1261] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.523793 31584 raft_consensus.cc:1261] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.523970 31324 raft_consensus.cc:1261] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.524711 2046 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.525895 2046 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.526176 2047 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.528404 31323 consensus_queue.cc:237] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:40.530040 1068 raft_consensus.cc:2932] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.530753 31583 raft_consensus.cc:1261] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.530570 31452 raft_consensus.cc:2932] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.530911 803 raft_consensus.cc:1261] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.532117 2090 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.532532 2090 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.532845 2072 raft_consensus.cc:2932] T 0ce9e7d46a6e475faaffb1ee26973db7 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.534862 2090 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.537721 672 raft_consensus.cc:1203] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:40.542315 31583 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.546597 803 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.547171 31044 catalog_manager.cc:5484] T 0ce9e7d46a6e475faaffb1ee26973db7 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 5 to 6, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.551853 31046 catalog_manager.cc:5484] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 reported cstate change: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.552351 31038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0ce9e7d46a6e475faaffb1ee26973db7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:40.554005 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9e964b80f5414d9ea3a5aea07b99d1d4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:40.556866 2017 heartbeater.cc:503] Master 127.23.104.254:43173 was elected leader, sending a full tablet report...
I20240503 02:13:40.557817 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e964b80f5414d9ea3a5aea07b99d1d4 with delete_type TABLET_DATA_TOMBSTONED (TS fbb7d16e67d94775b2881712419c0577 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41348
I20240503 02:13:40.558151 2051 tablet_replica.cc:330] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577: stopping tablet replica
I20240503 02:13:40.558383 2051 raft_consensus.cc:2229] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:40.558631 2051 raft_consensus.cc:2258] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:40.559836 2051 ts_tablet_manager.cc:1875] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:40.566318 2051 ts_tablet_manager.cc:1888] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:40.566437 2051 log.cc:1196] T 9e964b80f5414d9ea3a5aea07b99d1d4 P fbb7d16e67d94775b2881712419c0577: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/9e964b80f5414d9ea3a5aea07b99d1d4
I20240503 02:13:40.567289 31036 catalog_manager.cc:4830] TS fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101): tablet 9e964b80f5414d9ea3a5aea07b99d1d4 (table rebalance_test_table_1 [id=d3a62fd4d46c4ba082abbec8f97c84e0]) successfully deleted
I20240503 02:13:40.571909 31563 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce9e7d46a6e475faaffb1ee26973db7 with delete_type TABLET_DATA_TOMBSTONED (TS b92ff49cec804b1f801c11b64e3259b3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36462
I20240503 02:13:40.573276 2046 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.577286 2100 tablet_replica.cc:330] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3: stopping tablet replica
I20240503 02:13:40.577585 2100 raft_consensus.cc:2229] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:40.577839 2100 raft_consensus.cc:2258] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:40.579043 31324 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.579495 2100 ts_tablet_manager.cc:1875] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:40.582957 31584 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.584847 31048 catalog_manager.cc:5484] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.584671 673 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.591480 2100 ts_tablet_manager.cc:1888] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:40.591642 2100 log.cc:1196] T 0ce9e7d46a6e475faaffb1ee26973db7 P b92ff49cec804b1f801c11b64e3259b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/0ce9e7d46a6e475faaffb1ee26973db7
I20240503 02:13:40.592473 31452 consensus_queue.cc:237] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:40.593194 31039 catalog_manager.cc:4830] TS b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849): tablet 0ce9e7d46a6e475faaffb1ee26973db7 (table rebalance_test_table_6 [id=ad6bdc3b37e64f27824a7eacb47ca790]) successfully deleted
I20240503 02:13:40.594238 31584 raft_consensus.cc:1261] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.594240 673 raft_consensus.cc:1261] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.594750 2047 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.595011 1294 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.597736 2046 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.598240 31584 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.601858 31036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d9df1587246a4df19fda13f4be2b1177 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:40.602597 673 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:40.602877 31044 catalog_manager.cc:5484] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:40.607529 31303 tablet_service.cc:1514] Processing DeleteTablet for tablet d9df1587246a4df19fda13f4be2b1177 with delete_type TABLET_DATA_TOMBSTONED (TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55596
I20240503 02:13:40.607867 2127 tablet_replica.cc:330] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: stopping tablet replica
I20240503 02:13:40.608065 2127 raft_consensus.cc:2229] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:40.608292 2127 raft_consensus.cc:2258] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:40.609907 2127 ts_tablet_manager.cc:1875] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:40.615854 2127 ts_tablet_manager.cc:1888] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:40.616029 2127 log.cc:1196] T d9df1587246a4df19fda13f4be2b1177 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/d9df1587246a4df19fda13f4be2b1177
I20240503 02:13:40.616782 31037 catalog_manager.cc:4830] TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): tablet d9df1587246a4df19fda13f4be2b1177 (table rebalance_test_table_9 [id=3a5935f3f7924b5083fcfc4ee7015893]) successfully deleted
I20240503 02:13:40.628415 31324 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4dc576d8e72647c08736ae9508f6d27b"
dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49482
I20240503 02:13:40.628682 31324 raft_consensus.cc:595] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:40.660527 31452 consensus_queue.cc:237] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:40.662796 31584 raft_consensus.cc:1261] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:40.662997 31193 raft_consensus.cc:1261] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:40.663139 31389 consensus_peers.cc:485] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 -> Peer 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195): Couldn't send request to peer 05ef0a15decc468ea3a1af8828b0f630. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c425b03a1c64d7ba7a14901d63d7037. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:40.663383 2047 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:40.663679 2046 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:40.666128 2047 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:40.666736 31584 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:40.667441 31193 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:40.671837 31049 catalog_manager.cc:5484] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:40.674960 2175 raft_consensus.cc:979] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: : Instructing follower 0429a57df6ba4f92a73e779b04b05040 to start an election
I20240503 02:13:40.675127 2175 raft_consensus.cc:1067] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Signalling peer 0429a57df6ba4f92a73e779b04b05040 to start an election
I20240503 02:13:40.676074 1068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4dc576d8e72647c08736ae9508f6d27b"
dest_uuid: "0429a57df6ba4f92a73e779b04b05040"
from {username='slave'} at 127.23.104.194:52941
I20240503 02:13:40.676393 1068 raft_consensus.cc:482] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:40.676584 1068 raft_consensus.cc:3037] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:40.680691 1068 raft_consensus.cc:504] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:40.683005 1068 leader_election.cc:288] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 election: Requested vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:40.683451 31324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc576d8e72647c08736ae9508f6d27b" candidate_uuid: "0429a57df6ba4f92a73e779b04b05040" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
I20240503 02:13:40.683692 31324 raft_consensus.cc:3032] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:40.683809 31324 raft_consensus.cc:729] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:40.683876 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc576d8e72647c08736ae9508f6d27b" candidate_uuid: "0429a57df6ba4f92a73e779b04b05040" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:40.684145 31193 raft_consensus.cc:3037] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:40.684146 31324 consensus_queue.cc:260] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:40.684619 31324 raft_consensus.cc:3037] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:40.686836 31193 raft_consensus.cc:2454] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0429a57df6ba4f92a73e779b04b05040 in term 2.
I20240503 02:13:40.686952 31324 raft_consensus.cc:2454] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0429a57df6ba4f92a73e779b04b05040 in term 2.
I20240503 02:13:40.687317 31584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc576d8e72647c08736ae9508f6d27b" candidate_uuid: "0429a57df6ba4f92a73e779b04b05040" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:40.687542 31584 raft_consensus.cc:3037] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:40.687573 1002 leader_election.cc:302] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0429a57df6ba4f92a73e779b04b05040, b52c8be37a2241ff9dc555aa0cebd5e5, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:40.687981 1826 raft_consensus.cc:2781] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:40.688414 1826 raft_consensus.cc:686] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [term 2 LEADER]: Becoming Leader. State: Replica: 0429a57df6ba4f92a73e779b04b05040, State: Running, Role: LEADER
I20240503 02:13:40.688944 1826 consensus_queue.cc:237] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:40.690152 31584 raft_consensus.cc:2454] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0429a57df6ba4f92a73e779b04b05040 in term 2.
I20240503 02:13:40.693145 31050 catalog_manager.cc:5484] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 reported cstate change: term changed from 1 to 2, leader changed from cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) to 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200). New cstate: current_term: 2 leader_uuid: "0429a57df6ba4f92a73e779b04b05040" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:40.694427 31324 consensus_queue.cc:237] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
W20240503 02:13:40.696870 31258 consensus_peers.cc:485] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997): Couldn't send request to peer 4a3e3937b61447598fd76bc57f5d0939. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ce56ca4030a445db890303244caf308. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:40.696902 31584 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:40.697371 2175 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:40.697597 31193 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:40.698098 2175 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:40.700138 2090 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:40.701267 31193 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:40.703742 31584 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:40.705708 31044 catalog_manager.cc:5484] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:40.715694 31584 consensus_queue.cc:237] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:40.718854 31193 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:40.719478 2232 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:40.722358 2022 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:40.722783 31193 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:40.724736 31324 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:40.725261 1336 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:40.726626 31518 consensus_peers.cc:485] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 -> Peer 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543): Couldn't send request to peer 13d0f62249b34b01a3a47c6afe10e750. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a5406bce7014bc2a7a1d727cb4aa191. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:40.727505 31049 catalog_manager.cc:5484] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:40.728163 31324 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:40.742157 31324 consensus_queue.cc:237] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:40.744333 31584 raft_consensus.cc:1261] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:40.744419 31452 raft_consensus.cc:1261] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:40.745033 2090 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:40.745532 1357 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:40.746987 31260 consensus_peers.cc:485] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401): Couldn't send request to peer 0429a57df6ba4f92a73e779b04b05040. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb7e8fe39c024108be24bc4c144b247b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:40.748312 2090 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:40.749064 31584 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:40.749706 31452 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:40.753434 31050 catalog_manager.cc:5484] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:40.767220 2231 raft_consensus.cc:1050] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3: attempting to promote NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 to VOTER
I20240503 02:13:40.767253 31324 consensus_queue.cc:237] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:40.768101 2231 consensus_queue.cc:237] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:40.770134 31452 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:40.770380 31259 consensus_peers.cc:485] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195): Couldn't send request to peer 05ef0a15decc468ea3a1af8828b0f630. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c46912d2d4434d4a8c3fa0c3c19ba14b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:40.770411 31193 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:40.770689 1357 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:40.771010 2090 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:40.771847 31452 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.771857 936 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.772022 31324 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.772480 1336 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.773075 1336 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.773514 1336 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.776386 1357 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:40.777977 31193 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:40.781349 31044 catalog_manager.cc:5484] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:40.782903 31454 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:40.784700 1336 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.790323 31454 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.794528 31050 catalog_manager.cc:5484] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:40.795771 31324 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.796705 936 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.812346 31454 consensus_queue.cc:237] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:40.814443 31324 raft_consensus.cc:1261] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:40.814549 31193 raft_consensus.cc:1261] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:40.815033 2046 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:40.815337 1294 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:40.815654 31388 consensus_peers.cc:485] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 -> Peer 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543): Couldn't send request to peer 13d0f62249b34b01a3a47c6afe10e750. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c3d978cd00449b8b3edd4c43b657960. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:40.817759 1294 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:40.818435 31193 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:40.818789 31324 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:40.822768 31044 catalog_manager.cc:5484] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:40.853624 2175 raft_consensus.cc:1050] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 to VOTER
I20240503 02:13:40.854403 2175 consensus_queue.cc:237] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:40.860400 936 raft_consensus.cc:1261] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.860924 31454 raft_consensus.cc:1261] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.861622 31193 raft_consensus.cc:1261] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.861711 2090 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.862663 2090 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.863379 2175 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.876519 2175 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.879293 31454 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.880555 31193 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.882165 936 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.886189 31044 catalog_manager.cc:5484] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.892810 31324 consensus_queue.cc:237] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:40.894722 31454 raft_consensus.cc:1261] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.895226 936 raft_consensus.cc:1261] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.895241 2175 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.896387 2175 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.898618 1357 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.899250 936 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.901737 1336 raft_consensus.cc:1050] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3: attempting to promote NON_VOTER 0429a57df6ba4f92a73e779b04b05040 to VOTER
I20240503 02:13:40.902473 1336 consensus_queue.cc:237] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:40.904852 31049 catalog_manager.cc:5484] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 reported cstate change: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.905839 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8c681bdcf3074c54b1d533cf1f7d2b75 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:40.906920 31452 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.907481 2231 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.907833 31193 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.907887 1068 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.908640 2232 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.909266 31171 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c681bdcf3074c54b1d533cf1f7d2b75 with delete_type TABLET_DATA_TOMBSTONED (TS b52c8be37a2241ff9dc555aa0cebd5e5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47568
I20240503 02:13:40.909608 2039 tablet_replica.cc:330] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5: stopping tablet replica
I20240503 02:13:40.909791 2039 raft_consensus.cc:2229] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:40.909937 2175 raft_consensus.cc:1050] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 to VOTER
I20240503 02:13:40.910024 2039 raft_consensus.cc:2258] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:40.910903 2175 consensus_queue.cc:237] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:40.911145 2039 ts_tablet_manager.cc:1875] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:40.913872 2231 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.916678 31452 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:40.920557 2039 ts_tablet_manager.cc:1888] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:40.920751 2039 log.cc:1196] T 8c681bdcf3074c54b1d533cf1f7d2b75 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/8c681bdcf3074c54b1d533cf1f7d2b75
I20240503 02:13:40.921922 31038 catalog_manager.cc:4830] TS b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): tablet 8c681bdcf3074c54b1d533cf1f7d2b75 (table rebalance_test_table_4 [id=38897cba33374498b288d375d22641bd]) successfully deleted
I20240503 02:13:40.924968 803 raft_consensus.cc:1261] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.925107 31584 raft_consensus.cc:1261] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.926172 2174 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.926879 2174 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.932782 2022 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.934702 31452 raft_consensus.cc:1261] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:40.935233 1357 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:40.936323 1068 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.941591 31193 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.943560 31049 catalog_manager.cc:5484] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:40.951548 2090 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.957098 31454 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.959331 803 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.960151 31584 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.961701 31044 catalog_manager.cc:5484] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:40.968021 31324 consensus_queue.cc:237] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:40.969096 31452 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:40.970865 31584 raft_consensus.cc:1261] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.971400 2090 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.972124 803 raft_consensus.cc:1261] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:40.972784 2090 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:40.973755 1357 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.975883 804 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.977181 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7626c06dc67448f0b65f67b401d62c88 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:40.979051 31044 catalog_manager.cc:5484] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:40.980758 31583 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:40.982821 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet 7626c06dc67448f0b65f67b401d62c88 with delete_type TABLET_DATA_TOMBSTONED (TS fbb7d16e67d94775b2881712419c0577 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41348
I20240503 02:13:40.983173 2051 tablet_replica.cc:330] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577: stopping tablet replica
I20240503 02:13:40.983381 2051 raft_consensus.cc:2229] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:40.983690 2051 raft_consensus.cc:2258] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:40.985621 2051 ts_tablet_manager.cc:1875] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:40.994537 2051 ts_tablet_manager.cc:1888] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:40.994735 2051 log.cc:1196] T 7626c06dc67448f0b65f67b401d62c88 P fbb7d16e67d94775b2881712419c0577: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/7626c06dc67448f0b65f67b401d62c88
I20240503 02:13:40.995992 31036 catalog_manager.cc:4830] TS fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101): tablet 7626c06dc67448f0b65f67b401d62c88 (table rebalance_test_table_5 [id=f64903d295f2485983d7a34ee454f2b9]) successfully deleted
I20240503 02:13:41.062567 2341 ts_tablet_manager.cc:902] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630: Initiating tablet copy from peer fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:41.064203 2341 tablet_copy_client.cc:308] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:39101
I20240503 02:13:41.064900 31474 tablet_copy_service.cc:140] P fbb7d16e67d94775b2881712419c0577: Received BeginTabletCopySession request for tablet 1c425b03a1c64d7ba7a14901d63d7037 from peer 05ef0a15decc468ea3a1af8828b0f630 ({username='slave'} at 127.23.104.198:53527)
I20240503 02:13:41.065037 31474 tablet_copy_service.cc:161] P fbb7d16e67d94775b2881712419c0577: Beginning new tablet copy session on tablet 1c425b03a1c64d7ba7a14901d63d7037 from peer 05ef0a15decc468ea3a1af8828b0f630 at {username='slave'} at 127.23.104.198:53527: session id = 05ef0a15decc468ea3a1af8828b0f630-1c425b03a1c64d7ba7a14901d63d7037
I20240503 02:13:41.067461 31474 tablet_copy_source_session.cc:205] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:41.068596 2341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c425b03a1c64d7ba7a14901d63d7037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:41.074630 2341 tablet_copy_client.cc:783] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:41.074867 2341 tablet_copy_client.cc:647] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:41.076710 2345 tablet_copy_client.cc:1002] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:41.088963 31583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "17be048393974fdf9811c5b8f8a97579"
dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44000
I20240503 02:13:41.089175 31583 raft_consensus.cc:595] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:41.101666 2367 raft_consensus.cc:1050] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5: attempting to promote NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 to VOTER
I20240503 02:13:41.102803 2367 consensus_queue.cc:237] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:41.105585 673 raft_consensus.cc:1261] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:41.105738 31324 raft_consensus.cc:1261] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:41.106319 2179 consensus_queue.cc:1035] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:41.106815 2179 consensus_queue.cc:1035] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:41.114112 31583 raft_consensus.cc:1261] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:41.114696 2179 consensus_queue.cc:1035] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:41.120985 2366 raft_consensus.cc:2932] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:41.123751 31583 raft_consensus.cc:2932] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:41.125352 673 raft_consensus.cc:2932] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:41.127226 31324 raft_consensus.cc:2932] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:41.129177 31044 catalog_manager.cc:5484] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:41.135319 31193 consensus_queue.cc:237] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:41.137215 673 raft_consensus.cc:1261] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:41.137359 31583 raft_consensus.cc:1261] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:41.138062 2179 consensus_queue.cc:1035] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:41.138329 2367 consensus_queue.cc:1035] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:41.142019 31193 raft_consensus.cc:1261] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Refusing update from remote peer 0429a57df6ba4f92a73e779b04b05040: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:41.142572 1826 consensus_queue.cc:1035] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:41.142527 2179 raft_consensus.cc:2932] T f52b9a3c14a04815800753bda9505cee P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:41.143319 673 raft_consensus.cc:2932] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:41.146762 31038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f52b9a3c14a04815800753bda9505cee with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:41.147492 31583 raft_consensus.cc:2932] T f52b9a3c14a04815800753bda9505cee P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:41.148731 31050 catalog_manager.cc:5484] T f52b9a3c14a04815800753bda9505cee P 13d0f62249b34b01a3a47c6afe10e750 reported cstate change: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:41.155975 31303 tablet_service.cc:1514] Processing DeleteTablet for tablet f52b9a3c14a04815800753bda9505cee with delete_type TABLET_DATA_TOMBSTONED (TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55596
I20240503 02:13:41.156271 31584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "00e1dad5efda4e9eae55a8f9d80f0f96"
dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44014
I20240503 02:13:41.156466 31584 raft_consensus.cc:595] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:41.156716 2378 tablet_replica.cc:330] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: stopping tablet replica
I20240503 02:13:41.156968 2378 raft_consensus.cc:2229] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:41.157298 2378 raft_consensus.cc:2258] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:41.158190 2378 ts_tablet_manager.cc:1875] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:41.163414 2378 ts_tablet_manager.cc:1888] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:41.163606 2378 log.cc:1196] T f52b9a3c14a04815800753bda9505cee P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/f52b9a3c14a04815800753bda9505cee
I20240503 02:13:41.164760 31037 catalog_manager.cc:4830] TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): tablet f52b9a3c14a04815800753bda9505cee (table rebalance_test_table_8 [id=dc947496e186485092d5d621503a4e6f]) successfully deleted
I20240503 02:13:41.184880 2390 ts_tablet_manager.cc:902] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:41.188148 2390 tablet_copy_client.cc:308] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:41.189055 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet 6ce56ca4030a445db890303244caf308 from peer 4a3e3937b61447598fd76bc57f5d0939 ({username='slave'} at 127.23.104.199:34985)
I20240503 02:13:41.189193 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet 6ce56ca4030a445db890303244caf308 from peer 4a3e3937b61447598fd76bc57f5d0939 at {username='slave'} at 127.23.104.199:34985: session id = 4a3e3937b61447598fd76bc57f5d0939-6ce56ca4030a445db890303244caf308
I20240503 02:13:41.190806 31584 consensus_queue.cc:237] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:41.191850 31344 tablet_copy_source_session.cc:205] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:41.193163 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ce56ca4030a445db890303244caf308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:41.193997 31452 raft_consensus.cc:1261] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:41.194131 31193 raft_consensus.cc:1261] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:41.194548 2231 consensus_queue.cc:1035] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:41.194897 1336 consensus_queue.cc:1035] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:41.198442 2390 tablet_copy_client.cc:783] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:41.198665 2390 tablet_copy_client.cc:647] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 1 WAL segments...
W20240503 02:13:41.199951 31518 consensus_peers.cc:485] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 -> Peer 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543): Couldn't send request to peer 13d0f62249b34b01a3a47c6afe10e750. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2de731c4261f49088958535b8b7a65e9. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:41.199663 2396 raft_consensus.cc:2932] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:41.201431 2394 tablet_copy_client.cc:1002] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:41.205618 31049 catalog_manager.cc:5484] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:41.205574 31193 raft_consensus.cc:2932] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:41.214136 31454 raft_consensus.cc:2932] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:41.217622 31324 raft_consensus.cc:1261] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Refusing update from remote peer 0429a57df6ba4f92a73e779b04b05040: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:41.218191 1826 consensus_queue.cc:1035] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:41.223526 31584 raft_consensus.cc:1261] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Refusing update from remote peer 0429a57df6ba4f92a73e779b04b05040: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:41.224059 1826 consensus_queue.cc:1035] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:41.227393 31193 consensus_queue.cc:237] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:41.230199 31324 raft_consensus.cc:1261] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:41.230633 2411 ts_tablet_manager.cc:902] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750: Initiating tablet copy from peer fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:41.230803 2366 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:41.230973 31454 raft_consensus.cc:1261] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:41.232151 2366 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:41.233794 2232 raft_consensus.cc:979] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3: : Instructing follower cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 to start an election
I20240503 02:13:41.233937 2232 raft_consensus.cc:1067] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Signalling peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 to start an election
W20240503 02:13:41.234299 31129 consensus_peers.cc:485] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 -> Peer 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401): Couldn't send request to peer 0429a57df6ba4f92a73e779b04b05040. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4995796bdda64a7cbffa952a05a36673. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:41.235782 31323 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17be048393974fdf9811c5b8f8a97579"
dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
from {username='slave'} at 127.23.104.196:58559
I20240503 02:13:41.236073 31323 raft_consensus.cc:482] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:41.236210 31323 raft_consensus.cc:3037] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:41.239518 31323 raft_consensus.cc:504] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:41.242959 2366 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:41.245113 31584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17be048393974fdf9811c5b8f8a97579" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:41.245358 31584 raft_consensus.cc:3032] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:41.245479 31584 raft_consensus.cc:729] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Running, Role: LEADER
I20240503 02:13:41.246830 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17be048393974fdf9811c5b8f8a97579" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:41.246763 31452 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:41.247278 31454 raft_consensus.cc:3037] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:41.245829 31584 consensus_queue.cc:260] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:41.248183 31584 raft_consensus.cc:3037] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:41.250350 31454 raft_consensus.cc:2454] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 2.
I20240503 02:13:41.253499 31049 catalog_manager.cc:5484] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:41.261217 2418 ts_tablet_manager.cc:902] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:41.262270 31323 leader_election.cc:288] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 2 election: Requested vote from peers fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101), b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997)
I20240503 02:13:41.263604 936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17be048393974fdf9811c5b8f8a97579" candidate_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4a3e3937b61447598fd76bc57f5d0939"
I20240503 02:13:41.264034 936 raft_consensus.cc:3037] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:41.266240 31584 raft_consensus.cc:2454] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 2.
I20240503 02:13:41.266825 31260 leader_election.cc:302] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: b92ff49cec804b1f801c11b64e3259b3, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:41.267328 2090 raft_consensus.cc:2781] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:41.267486 2090 raft_consensus.cc:686] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 LEADER]: Becoming Leader. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:41.267895 2090 consensus_queue.cc:237] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:41.271706 31049 catalog_manager.cc:5484] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: term changed from 1 to 2, leader changed from b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) to cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194). New cstate: current_term: 2 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:41.277707 936 raft_consensus.cc:2454] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 in term 2.
I20240503 02:13:41.282186 31323 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:41.288120 2411 tablet_copy_client.cc:308] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:39101
I20240503 02:13:41.288808 31474 tablet_copy_service.cc:140] P fbb7d16e67d94775b2881712419c0577: Received BeginTabletCopySession request for tablet 6c3d978cd00449b8b3edd4c43b657960 from peer 13d0f62249b34b01a3a47c6afe10e750 ({username='slave'} at 127.23.104.197:57809)
I20240503 02:13:41.289028 31474 tablet_copy_service.cc:161] P fbb7d16e67d94775b2881712419c0577: Beginning new tablet copy session on tablet 6c3d978cd00449b8b3edd4c43b657960 from peer 13d0f62249b34b01a3a47c6afe10e750 at {username='slave'} at 127.23.104.197:57809: session id = 13d0f62249b34b01a3a47c6afe10e750-6c3d978cd00449b8b3edd4c43b657960
I20240503 02:13:41.292951 31474 tablet_copy_source_session.cc:205] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:41.294466 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3d978cd00449b8b3edd4c43b657960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:41.296196 2418 tablet_copy_client.cc:308] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:41.296833 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet c46912d2d4434d4a8c3fa0c3c19ba14b from peer 05ef0a15decc468ea3a1af8828b0f630 ({username='slave'} at 127.23.104.198:42437)
I20240503 02:13:41.297022 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet c46912d2d4434d4a8c3fa0c3c19ba14b from peer 05ef0a15decc468ea3a1af8828b0f630 at {username='slave'} at 127.23.104.198:42437: session id = 05ef0a15decc468ea3a1af8828b0f630-c46912d2d4434d4a8c3fa0c3c19ba14b
I20240503 02:13:41.299819 31344 tablet_copy_source_session.cc:205] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:41.301192 2411 tablet_copy_client.cc:783] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:41.301492 2411 tablet_copy_client.cc:647] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:41.302103 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46912d2d4434d4a8c3fa0c3c19ba14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:41.302551 31584 consensus_queue.cc:237] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:41.303083 2424 tablet_copy_client.cc:1002] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:41.305020 31193 raft_consensus.cc:1261] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:41.305296 31452 raft_consensus.cc:1261] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:41.305575 2232 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:41.305832 31518 consensus_peers.cc:485] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 -> Peer 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543): Couldn't send request to peer 13d0f62249b34b01a3a47c6afe10e750. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d8d17ee831446ae8d4a0417ede894b8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:41.306056 2231 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:41.308032 2418 tablet_copy_client.cc:783] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:41.308282 2418 tablet_copy_client.cc:647] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:41.308789 2232 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:41.309540 31193 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:41.309675 31452 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:41.315270 31050 catalog_manager.cc:5484] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:41.315896 2427 tablet_copy_client.cc:1002] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:41.316376 2431 ts_tablet_manager.cc:902] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750: Initiating tablet copy from peer b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:41.325956 2431 tablet_copy_client.cc:308] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:39849
I20240503 02:13:41.326599 31604 tablet_copy_service.cc:140] P b92ff49cec804b1f801c11b64e3259b3: Received BeginTabletCopySession request for tablet 8a5406bce7014bc2a7a1d727cb4aa191 from peer 13d0f62249b34b01a3a47c6afe10e750 ({username='slave'} at 127.23.104.197:53893)
I20240503 02:13:41.326752 31604 tablet_copy_service.cc:161] P b92ff49cec804b1f801c11b64e3259b3: Beginning new tablet copy session on tablet 8a5406bce7014bc2a7a1d727cb4aa191 from peer 13d0f62249b34b01a3a47c6afe10e750 at {username='slave'} at 127.23.104.197:53893: session id = 13d0f62249b34b01a3a47c6afe10e750-8a5406bce7014bc2a7a1d727cb4aa191
I20240503 02:13:41.328867 31604 tablet_copy_source_session.cc:205] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:41.330550 2431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a5406bce7014bc2a7a1d727cb4aa191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:41.336997 2431 tablet_copy_client.cc:783] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:41.337273 2431 tablet_copy_client.cc:647] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:41.338798 2440 tablet_copy_client.cc:1002] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:41.364225 2461 ts_tablet_manager.cc:902] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:41.368887 2461 tablet_copy_client.cc:308] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:41.369534 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet cb7e8fe39c024108be24bc4c144b247b from peer 0429a57df6ba4f92a73e779b04b05040 ({username='slave'} at 127.23.104.200:44199)
I20240503 02:13:41.369650 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet cb7e8fe39c024108be24bc4c144b247b from peer 0429a57df6ba4f92a73e779b04b05040 at {username='slave'} at 127.23.104.200:44199: session id = 0429a57df6ba4f92a73e779b04b05040-cb7e8fe39c024108be24bc4c144b247b
I20240503 02:13:41.371515 31344 tablet_copy_source_session.cc:205] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:41.372553 2461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7e8fe39c024108be24bc4c144b247b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:41.376649 2461 tablet_copy_client.cc:783] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:41.376866 2461 tablet_copy_client.cc:647] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:41.378340 2469 tablet_copy_client.cc:1002] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:41.432605 1336 raft_consensus.cc:979] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3: : Instructing follower b52c8be37a2241ff9dc555aa0cebd5e5 to start an election
I20240503 02:13:41.432766 1336 raft_consensus.cc:1067] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Signalling peer b52c8be37a2241ff9dc555aa0cebd5e5 to start an election
I20240503 02:13:41.433583 31193 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00e1dad5efda4e9eae55a8f9d80f0f96"
dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
from {username='slave'} at 127.23.104.196:48301
I20240503 02:13:41.433779 31193 raft_consensus.cc:482] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:41.433895 31193 raft_consensus.cc:3037] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:41.435932 31193 raft_consensus.cc:504] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:41.437110 31584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e1dad5efda4e9eae55a8f9d80f0f96" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:41.437331 31584 raft_consensus.cc:3032] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:41.437409 31584 raft_consensus.cc:729] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Running, Role: LEADER
I20240503 02:13:41.437690 31584 consensus_queue.cc:260] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:41.438103 31584 raft_consensus.cc:3037] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:41.440408 31452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e1dad5efda4e9eae55a8f9d80f0f96" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:41.441350 31452 raft_consensus.cc:3037] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:41.443522 31452 raft_consensus.cc:2454] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 2.
I20240503 02:13:41.444375 31193 leader_election.cc:288] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 2 election: Requested vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101), 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401)
I20240503 02:13:41.445025 31584 raft_consensus.cc:2454] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 2.
I20240503 02:13:41.445156 1068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e1dad5efda4e9eae55a8f9d80f0f96" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "0429a57df6ba4f92a73e779b04b05040"
I20240503 02:13:41.445397 31129 leader_election.cc:302] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, b92ff49cec804b1f801c11b64e3259b3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:41.445456 1068 raft_consensus.cc:3037] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:41.445653 2367 raft_consensus.cc:2781] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:41.445809 2367 raft_consensus.cc:686] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 LEADER]: Becoming Leader. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Running, Role: LEADER
I20240503 02:13:41.446080 2367 consensus_queue.cc:237] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:41.450214 1068 raft_consensus.cc:2454] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 2.
I20240503 02:13:41.450083 31049 catalog_manager.cc:5484] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: term changed from 1 to 2, leader changed from b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) to b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193). New cstate: current_term: 2 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:41.479674 31324 consensus_queue.cc:237] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:41.481711 31193 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:41.481945 31452 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:41.482231 1357 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:41.482591 31258 consensus_peers.cc:485] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543): Couldn't send request to peer 13d0f62249b34b01a3a47c6afe10e750. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c91a7b003ed9411aba65a46d00c7d95b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:41.482618 2518 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:41.485109 1357 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:41.485678 31193 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:41.486014 31452 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:41.490366 31048 catalog_manager.cc:5484] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:41.507898 31452 consensus_queue.cc:237] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:41.510118 31324 raft_consensus.cc:1261] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:41.510275 31193 raft_consensus.cc:1261] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:41.510841 1294 consensus_queue.cc:1035] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:41.511188 2501 consensus_queue.cc:1035] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:41.514833 1294 raft_consensus.cc:2932] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:41.515477 31193 raft_consensus.cc:2932] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:41.516162 31324 raft_consensus.cc:2932] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
W20240503 02:13:41.520227 31390 consensus_peers.cc:485] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 -> Peer 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401): Couldn't send request to peer 0429a57df6ba4f92a73e779b04b05040. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f8e7774779e47ba9d426ac00ad8be62. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:41.521919 31050 catalog_manager.cc:5484] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index -1 to 4, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:41.646431 2576 ts_tablet_manager.cc:902] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040: Initiating tablet copy from peer b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:41.654345 2576 tablet_copy_client.cc:308] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.193:44019
I20240503 02:13:41.655066 31213 tablet_copy_service.cc:140] P b52c8be37a2241ff9dc555aa0cebd5e5: Received BeginTabletCopySession request for tablet 4995796bdda64a7cbffa952a05a36673 from peer 0429a57df6ba4f92a73e779b04b05040 ({username='slave'} at 127.23.104.200:44419)
I20240503 02:13:41.655206 31213 tablet_copy_service.cc:161] P b52c8be37a2241ff9dc555aa0cebd5e5: Beginning new tablet copy session on tablet 4995796bdda64a7cbffa952a05a36673 from peer 0429a57df6ba4f92a73e779b04b05040 at {username='slave'} at 127.23.104.200:44419: session id = 0429a57df6ba4f92a73e779b04b05040-4995796bdda64a7cbffa952a05a36673
I20240503 02:13:41.657464 31213 tablet_copy_source_session.cc:205] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:41.658835 2576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4995796bdda64a7cbffa952a05a36673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:41.661506 31452 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:41.662242 1357 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:41.663306 2576 tablet_copy_client.cc:783] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:41.663517 2576 tablet_copy_client.cc:647] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:41.665700 2580 tablet_copy_client.cc:1002] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:41.730744 936 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:41.731205 1357 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:41.734573 31584 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:41.735142 2090 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:41.737946 31324 consensus_queue.cc:237] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:41.740041 31452 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:41.740829 2090 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:41.743254 1357 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:41.747242 31452 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:41.752425 1068 consensus_queue.cc:237] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:41.754303 31583 raft_consensus.cc:1261] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Refusing update from remote peer 0429a57df6ba4f92a73e779b04b05040: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:41.754962 31193 raft_consensus.cc:1261] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Refusing update from remote peer 0429a57df6ba4f92a73e779b04b05040: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:41.755273 2407 consensus_queue.cc:1035] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:41.755666 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 17be048393974fdf9811c5b8f8a97579 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:41.756601 31049 catalog_manager.cc:5484] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 5 to 7, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New cstate: current_term: 2 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:41.757331 2407 consensus_queue.cc:1035] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:41.758446 2407 raft_consensus.cc:2932] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:41.760574 31563 tablet_service.cc:1514] Processing DeleteTablet for tablet 17be048393974fdf9811c5b8f8a97579 with delete_type TABLET_DATA_TOMBSTONED (TS b92ff49cec804b1f801c11b64e3259b3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36462
I20240503 02:13:41.762351 31583 raft_consensus.cc:2932] T 4dc576d8e72647c08736ae9508f6d27b P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:41.762532 31193 raft_consensus.cc:2932] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:41.765138 2612 tablet_replica.cc:330] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3: stopping tablet replica
I20240503 02:13:41.765388 2612 raft_consensus.cc:2229] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:41.765686 2612 raft_consensus.cc:2258] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:41.766791 2612 ts_tablet_manager.cc:1875] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:41.768054 31039 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4dc576d8e72647c08736ae9508f6d27b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:41.769407 31049 catalog_manager.cc:5484] T 4dc576d8e72647c08736ae9508f6d27b P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index 5 to 7, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New cstate: current_term: 2 leader_uuid: "0429a57df6ba4f92a73e779b04b05040" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:41.773116 2612 ts_tablet_manager.cc:1888] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:13:41.773254 2612 log.cc:1196] T 17be048393974fdf9811c5b8f8a97579 P b92ff49cec804b1f801c11b64e3259b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/17be048393974fdf9811c5b8f8a97579
I20240503 02:13:41.773320 31303 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc576d8e72647c08736ae9508f6d27b with delete_type TABLET_DATA_TOMBSTONED (TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55596
I20240503 02:13:41.773979 31039 catalog_manager.cc:4830] TS b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849): tablet 17be048393974fdf9811c5b8f8a97579 (table rebalance_test_table_3 [id=e0a6e74497494a20be21d627357e4eb0]) successfully deleted
I20240503 02:13:41.774070 2614 tablet_replica.cc:330] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: stopping tablet replica
I20240503 02:13:41.774322 2614 raft_consensus.cc:2229] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:41.774498 2614 raft_consensus.cc:2258] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:41.775182 2614 ts_tablet_manager.cc:1875] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:41.776620 935 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:41.777356 1357 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:41.778776 935 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:41.780253 2614 ts_tablet_manager.cc:1888] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:13:41.780414 2614 log.cc:1196] T 4dc576d8e72647c08736ae9508f6d27b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/4dc576d8e72647c08736ae9508f6d27b
I20240503 02:13:41.781667 31037 catalog_manager.cc:4830] TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): tablet 4dc576d8e72647c08736ae9508f6d27b (table rebalance_test_table_4 [id=38897cba33374498b288d375d22641bd]) successfully deleted
I20240503 02:13:41.792670 2615 ts_tablet_manager.cc:902] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750: Initiating tablet copy from peer b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:41.793869 2615 tablet_copy_client.cc:308] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:39849
I20240503 02:13:41.794471 31604 tablet_copy_service.cc:140] P b92ff49cec804b1f801c11b64e3259b3: Received BeginTabletCopySession request for tablet 8d8d17ee831446ae8d4a0417ede894b8 from peer 13d0f62249b34b01a3a47c6afe10e750 ({username='slave'} at 127.23.104.197:53893)
I20240503 02:13:41.794617 31604 tablet_copy_service.cc:161] P b92ff49cec804b1f801c11b64e3259b3: Beginning new tablet copy session on tablet 8d8d17ee831446ae8d4a0417ede894b8 from peer 13d0f62249b34b01a3a47c6afe10e750 at {username='slave'} at 127.23.104.197:53893: session id = 13d0f62249b34b01a3a47c6afe10e750-8d8d17ee831446ae8d4a0417ede894b8
I20240503 02:13:41.796834 31604 tablet_copy_source_session.cc:205] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:41.797890 2615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8d17ee831446ae8d4a0417ede894b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:41.802045 2615 tablet_copy_client.cc:783] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:41.802250 2615 tablet_copy_client.cc:647] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:41.803943 2619 tablet_copy_client.cc:1002] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:41.818392 2620 ts_tablet_manager.cc:902] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750: Initiating tablet copy from peer b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:41.819741 2620 tablet_copy_client.cc:308] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:39849
I20240503 02:13:41.820380 31604 tablet_copy_service.cc:140] P b92ff49cec804b1f801c11b64e3259b3: Received BeginTabletCopySession request for tablet 2de731c4261f49088958535b8b7a65e9 from peer 13d0f62249b34b01a3a47c6afe10e750 ({username='slave'} at 127.23.104.197:53893)
I20240503 02:13:41.820503 31604 tablet_copy_service.cc:161] P b92ff49cec804b1f801c11b64e3259b3: Beginning new tablet copy session on tablet 2de731c4261f49088958535b8b7a65e9 from peer 13d0f62249b34b01a3a47c6afe10e750 at {username='slave'} at 127.23.104.197:53893: session id = 13d0f62249b34b01a3a47c6afe10e750-2de731c4261f49088958535b8b7a65e9
I20240503 02:13:41.822353 31604 tablet_copy_source_session.cc:205] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:41.823339 2620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de731c4261f49088958535b8b7a65e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:41.827210 2620 tablet_copy_client.cc:783] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:41.827436 2620 tablet_copy_client.cc:647] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:41.828778 2624 tablet_copy_client.cc:1002] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:41.879601 2625 ts_tablet_manager.cc:902] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:41.880949 2625 tablet_copy_client.cc:308] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:41.881541 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet c91a7b003ed9411aba65a46d00c7d95b from peer 13d0f62249b34b01a3a47c6afe10e750 ({username='slave'} at 127.23.104.197:33983)
I20240503 02:13:41.881705 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet c91a7b003ed9411aba65a46d00c7d95b from peer 13d0f62249b34b01a3a47c6afe10e750 at {username='slave'} at 127.23.104.197:33983: session id = 13d0f62249b34b01a3a47c6afe10e750-c91a7b003ed9411aba65a46d00c7d95b
I20240503 02:13:41.883551 31344 tablet_copy_source_session.cc:205] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:41.884683 2625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91a7b003ed9411aba65a46d00c7d95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:41.889700 2625 tablet_copy_client.cc:783] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:41.889950 2625 tablet_copy_client.cc:647] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:41.891412 2629 tablet_copy_client.cc:1002] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:41.897462 2630 ts_tablet_manager.cc:902] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040: Initiating tablet copy from peer fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:41.899247 2630 tablet_copy_client.cc:308] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:39101
I20240503 02:13:41.903223 1068 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:41.903838 2367 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:41.907079 31474 tablet_copy_service.cc:140] P fbb7d16e67d94775b2881712419c0577: Received BeginTabletCopySession request for tablet 6f8e7774779e47ba9d426ac00ad8be62 from peer 0429a57df6ba4f92a73e779b04b05040 ({username='slave'} at 127.23.104.200:36057)
I20240503 02:13:41.907245 31474 tablet_copy_service.cc:161] P fbb7d16e67d94775b2881712419c0577: Beginning new tablet copy session on tablet 6f8e7774779e47ba9d426ac00ad8be62 from peer 0429a57df6ba4f92a73e779b04b05040 at {username='slave'} at 127.23.104.200:36057: session id = 0429a57df6ba4f92a73e779b04b05040-6f8e7774779e47ba9d426ac00ad8be62
I20240503 02:13:41.909750 31474 tablet_copy_source_session.cc:205] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:41.910970 2630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8e7774779e47ba9d426ac00ad8be62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:41.916869 2630 tablet_copy_client.cc:783] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:41.917100 2630 tablet_copy_client.cc:647] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:41.918681 2634 tablet_copy_client.cc:1002] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:42.021884 31583 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:42.022495 2367 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:42.042621 31452 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:42.043251 2707 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:42.577143 2341 tablet_copy_client.cc:523] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:42.579738 2341 tablet_bootstrap.cc:492] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap starting.
I20240503 02:13:42.632591 2341 tablet_bootstrap.cc:492] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:42.633097 2341 tablet_bootstrap.cc:492] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap complete.
I20240503 02:13:42.633445 2341 ts_tablet_manager.cc:1367] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630: Time spent bootstrapping tablet: real 0.054s user 0.051s sys 0.004s
I20240503 02:13:42.634372 2341 raft_consensus.cc:348] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:42.634714 2341 raft_consensus.cc:729] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05ef0a15decc468ea3a1af8828b0f630, State: Initialized, Role: LEARNER
I20240503 02:13:42.634966 2341 consensus_queue.cc:260] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:42.637280 2341 ts_tablet_manager.cc:1398] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:13:42.637852 31191 consensus_queue.cc:237] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:42.638724 31474 tablet_copy_service.cc:342] P fbb7d16e67d94775b2881712419c0577: Request end of tablet copy session 05ef0a15decc468ea3a1af8828b0f630-1c425b03a1c64d7ba7a14901d63d7037 received from {username='slave'} at 127.23.104.198:53527
I20240503 02:13:42.638911 31474 tablet_copy_service.cc:433] P fbb7d16e67d94775b2881712419c0577: ending tablet copy session 05ef0a15decc468ea3a1af8828b0f630-1c425b03a1c64d7ba7a14901d63d7037 on tablet 1c425b03a1c64d7ba7a14901d63d7037 with peer 05ef0a15decc468ea3a1af8828b0f630
I20240503 02:13:42.640561 31452 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:42.640580 1068 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:42.642755 2833 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:42.644125 2179 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:42.646198 2833 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:42.647657 31452 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:42.647920 1068 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:42.649956 31038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00e1dad5efda4e9eae55a8f9d80f0f96 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:42.652595 31048 catalog_manager.cc:5484] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index 5 to 7, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New cstate: current_term: 2 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:42.656809 31563 tablet_service.cc:1514] Processing DeleteTablet for tablet 00e1dad5efda4e9eae55a8f9d80f0f96 with delete_type TABLET_DATA_TOMBSTONED (TS b92ff49cec804b1f801c11b64e3259b3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36462
I20240503 02:13:42.657676 2838 tablet_replica.cc:330] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3: stopping tablet replica
I20240503 02:13:42.657887 2838 raft_consensus.cc:2229] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:42.658138 2838 raft_consensus.cc:2258] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:42.659054 2838 ts_tablet_manager.cc:1875] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:42.663374 2838 ts_tablet_manager.cc:1888] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:13:42.663499 2838 log.cc:1196] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b92ff49cec804b1f801c11b64e3259b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/00e1dad5efda4e9eae55a8f9d80f0f96
I20240503 02:13:42.664244 31039 catalog_manager.cc:4830] TS b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849): tablet 00e1dad5efda4e9eae55a8f9d80f0f96 (table rebalance_test_table_0 [id=cc3c62c772ec4610892f67e8c6d52379]) successfully deleted
I20240503 02:13:42.701819 2390 tablet_copy_client.cc:523] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:42.705063 2390 tablet_bootstrap.cc:492] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap starting.
I20240503 02:13:42.746434 2390 tablet_bootstrap.cc:492] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:42.747661 2390 tablet_bootstrap.cc:492] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap complete.
I20240503 02:13:42.747963 2390 ts_tablet_manager.cc:1367] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939: Time spent bootstrapping tablet: real 0.043s user 0.040s sys 0.000s
I20240503 02:13:42.748576 2390 raft_consensus.cc:348] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:42.748786 2390 raft_consensus.cc:729] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a3e3937b61447598fd76bc57f5d0939, State: Initialized, Role: LEARNER
I20240503 02:13:42.749022 2390 consensus_queue.cc:260] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:42.756541 2390 ts_tablet_manager.cc:1398] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20240503 02:13:42.757297 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 4a3e3937b61447598fd76bc57f5d0939-6ce56ca4030a445db890303244caf308 received from {username='slave'} at 127.23.104.199:34985
I20240503 02:13:42.757457 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 4a3e3937b61447598fd76bc57f5d0939-6ce56ca4030a445db890303244caf308 on tablet 6ce56ca4030a445db890303244caf308 with peer 4a3e3937b61447598fd76bc57f5d0939
I20240503 02:13:42.803467 2411 tablet_copy_client.cc:523] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:42.806550 2411 tablet_bootstrap.cc:492] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap starting.
I20240503 02:13:42.816419 2418 tablet_copy_client.cc:523] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:42.819717 2418 tablet_bootstrap.cc:492] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap starting.
I20240503 02:13:42.838187 804 raft_consensus.cc:1203] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:42.841370 2431 tablet_copy_client.cc:523] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:42.843858 2431 tablet_bootstrap.cc:492] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap starting.
I20240503 02:13:42.866315 2411 tablet_bootstrap.cc:492] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:42.866786 2411 tablet_bootstrap.cc:492] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap complete.
I20240503 02:13:42.867182 2411 ts_tablet_manager.cc:1367] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750: Time spent bootstrapping tablet: real 0.061s user 0.038s sys 0.001s
I20240503 02:13:42.868341 2411 raft_consensus.cc:348] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:42.868649 2411 raft_consensus.cc:729] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d0f62249b34b01a3a47c6afe10e750, State: Initialized, Role: LEARNER
I20240503 02:13:42.868914 2411 consensus_queue.cc:260] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:42.874344 2411 ts_tablet_manager.cc:1398] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240503 02:13:42.876132 31474 tablet_copy_service.cc:342] P fbb7d16e67d94775b2881712419c0577: Request end of tablet copy session 13d0f62249b34b01a3a47c6afe10e750-6c3d978cd00449b8b3edd4c43b657960 received from {username='slave'} at 127.23.104.197:57809
I20240503 02:13:42.876271 31474 tablet_copy_service.cc:433] P fbb7d16e67d94775b2881712419c0577: ending tablet copy session 13d0f62249b34b01a3a47c6afe10e750-6c3d978cd00449b8b3edd4c43b657960 on tablet 6c3d978cd00449b8b3edd4c43b657960 with peer 13d0f62249b34b01a3a47c6afe10e750
I20240503 02:13:42.878458 2418 tablet_bootstrap.cc:492] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:42.878911 2418 tablet_bootstrap.cc:492] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap complete.
I20240503 02:13:42.879232 2418 ts_tablet_manager.cc:1367] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630: Time spent bootstrapping tablet: real 0.060s user 0.047s sys 0.000s
I20240503 02:13:42.879827 2461 tablet_copy_client.cc:523] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:42.881220 2418 raft_consensus.cc:348] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:42.881531 2418 raft_consensus.cc:729] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05ef0a15decc468ea3a1af8828b0f630, State: Initialized, Role: LEARNER
I20240503 02:13:42.881764 2418 consensus_queue.cc:260] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:42.882383 2418 ts_tablet_manager.cc:1398] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240503 02:13:42.883634 2461 tablet_bootstrap.cc:492] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040: Bootstrap starting.
I20240503 02:13:42.884245 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 05ef0a15decc468ea3a1af8828b0f630-c46912d2d4434d4a8c3fa0c3c19ba14b received from {username='slave'} at 127.23.104.198:42437
I20240503 02:13:42.884387 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 05ef0a15decc468ea3a1af8828b0f630-c46912d2d4434d4a8c3fa0c3c19ba14b on tablet c46912d2d4434d4a8c3fa0c3c19ba14b with peer 05ef0a15decc468ea3a1af8828b0f630
I20240503 02:13:42.898067 935 raft_consensus.cc:1203] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:42.911077 2431 tablet_bootstrap.cc:492] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:42.911461 2431 tablet_bootstrap.cc:492] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap complete.
I20240503 02:13:42.911702 2431 ts_tablet_manager.cc:1367] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750: Time spent bootstrapping tablet: real 0.068s user 0.043s sys 0.004s
I20240503 02:13:42.912426 2431 raft_consensus.cc:348] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:42.912665 2431 raft_consensus.cc:729] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d0f62249b34b01a3a47c6afe10e750, State: Initialized, Role: LEARNER
I20240503 02:13:42.912873 2431 consensus_queue.cc:260] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:42.913473 2431 ts_tablet_manager.cc:1398] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 02:13:42.914158 31604 tablet_copy_service.cc:342] P b92ff49cec804b1f801c11b64e3259b3: Request end of tablet copy session 13d0f62249b34b01a3a47c6afe10e750-8a5406bce7014bc2a7a1d727cb4aa191 received from {username='slave'} at 127.23.104.197:53893
I20240503 02:13:42.914307 31604 tablet_copy_service.cc:433] P b92ff49cec804b1f801c11b64e3259b3: ending tablet copy session 13d0f62249b34b01a3a47c6afe10e750-8a5406bce7014bc2a7a1d727cb4aa191 on tablet 8a5406bce7014bc2a7a1d727cb4aa191 with peer 13d0f62249b34b01a3a47c6afe10e750
I20240503 02:13:42.927955 2461 tablet_bootstrap.cc:492] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:42.928382 2461 tablet_bootstrap.cc:492] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040: Bootstrap complete.
I20240503 02:13:42.928653 2461 ts_tablet_manager.cc:1367] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040: Time spent bootstrapping tablet: real 0.045s user 0.031s sys 0.008s
I20240503 02:13:42.929533 2461 raft_consensus.cc:348] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:42.929811 2461 raft_consensus.cc:729] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0429a57df6ba4f92a73e779b04b05040, State: Initialized, Role: LEARNER
I20240503 02:13:42.930038 2461 consensus_queue.cc:260] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:42.930734 2461 ts_tablet_manager.cc:1398] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:13:42.931525 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 0429a57df6ba4f92a73e779b04b05040-cb7e8fe39c024108be24bc4c144b247b received from {username='slave'} at 127.23.104.200:44199
I20240503 02:13:42.931654 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 0429a57df6ba4f92a73e779b04b05040-cb7e8fe39c024108be24bc4c144b247b on tablet cb7e8fe39c024108be24bc4c144b247b with peer 0429a57df6ba4f92a73e779b04b05040
I20240503 02:13:42.993747 672 raft_consensus.cc:1203] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:43.166052 2576 tablet_copy_client.cc:523] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:43.168658 2576 tablet_bootstrap.cc:492] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap starting.
I20240503 02:13:43.209645 2576 tablet_bootstrap.cc:492] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:43.210189 2576 tablet_bootstrap.cc:492] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap complete.
I20240503 02:13:43.210541 2576 ts_tablet_manager.cc:1367] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040: Time spent bootstrapping tablet: real 0.042s user 0.034s sys 0.004s
I20240503 02:13:43.211558 2576 raft_consensus.cc:348] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:43.211947 2576 raft_consensus.cc:729] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0429a57df6ba4f92a73e779b04b05040, State: Initialized, Role: LEARNER
I20240503 02:13:43.212208 2576 consensus_queue.cc:260] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:43.212991 2576 ts_tablet_manager.cc:1398] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:43.213742 31213 tablet_copy_service.cc:342] P b52c8be37a2241ff9dc555aa0cebd5e5: Request end of tablet copy session 0429a57df6ba4f92a73e779b04b05040-4995796bdda64a7cbffa952a05a36673 received from {username='slave'} at 127.23.104.200:44419
I20240503 02:13:43.213872 31213 tablet_copy_service.cc:433] P b52c8be37a2241ff9dc555aa0cebd5e5: ending tablet copy session 0429a57df6ba4f92a73e779b04b05040-4995796bdda64a7cbffa952a05a36673 on tablet 4995796bdda64a7cbffa952a05a36673 with peer 0429a57df6ba4f92a73e779b04b05040
I20240503 02:13:43.227720 1068 raft_consensus.cc:1203] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:43.234767 804 raft_consensus.cc:1203] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:43.304286 2615 tablet_copy_client.cc:523] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:43.306828 2615 tablet_bootstrap.cc:492] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap starting.
I20240503 02:13:43.329068 2620 tablet_copy_client.cc:523] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:43.332073 2620 tablet_bootstrap.cc:492] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap starting.
I20240503 02:13:43.338150 2615 tablet_bootstrap.cc:492] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:43.338549 2615 tablet_bootstrap.cc:492] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap complete.
I20240503 02:13:43.338801 2615 ts_tablet_manager.cc:1367] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.005s
I20240503 02:13:43.339371 2615 raft_consensus.cc:348] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:43.339597 2615 raft_consensus.cc:729] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d0f62249b34b01a3a47c6afe10e750, State: Initialized, Role: LEARNER
I20240503 02:13:43.339848 2615 consensus_queue.cc:260] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:43.340495 2615 ts_tablet_manager.cc:1398] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:43.341185 31604 tablet_copy_service.cc:342] P b92ff49cec804b1f801c11b64e3259b3: Request end of tablet copy session 13d0f62249b34b01a3a47c6afe10e750-8d8d17ee831446ae8d4a0417ede894b8 received from {username='slave'} at 127.23.104.197:53893
I20240503 02:13:43.341295 31604 tablet_copy_service.cc:433] P b92ff49cec804b1f801c11b64e3259b3: ending tablet copy session 13d0f62249b34b01a3a47c6afe10e750-8d8d17ee831446ae8d4a0417ede894b8 on tablet 8d8d17ee831446ae8d4a0417ede894b8 with peer 13d0f62249b34b01a3a47c6afe10e750
I20240503 02:13:43.358670 1294 raft_consensus.cc:1050] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577: attempting to promote NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 to VOTER
I20240503 02:13:43.359503 1294 consensus_queue.cc:237] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:43.362452 804 raft_consensus.cc:1261] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.363051 2963 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.363152 31191 raft_consensus.cc:1261] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.363677 31584 raft_consensus.cc:1261] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.364414 2501 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.366266 2501 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.367650 672 raft_consensus.cc:1203] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:43.375272 2620 tablet_bootstrap.cc:492] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:43.374809 3025 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:43.375720 2620 tablet_bootstrap.cc:492] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap complete.
I20240503 02:13:43.375506 31191 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:43.376015 2620 ts_tablet_manager.cc:1367] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750: Time spent bootstrapping tablet: real 0.044s user 0.040s sys 0.000s
I20240503 02:13:43.376741 2620 raft_consensus.cc:348] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:43.377014 2620 raft_consensus.cc:729] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d0f62249b34b01a3a47c6afe10e750, State: Initialized, Role: LEARNER
I20240503 02:13:43.377229 2620 consensus_queue.cc:260] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:43.378484 2620 ts_tablet_manager.cc:1398] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:43.379154 31604 tablet_copy_service.cc:342] P b92ff49cec804b1f801c11b64e3259b3: Request end of tablet copy session 13d0f62249b34b01a3a47c6afe10e750-2de731c4261f49088958535b8b7a65e9 received from {username='slave'} at 127.23.104.197:53893
I20240503 02:13:43.379300 31604 tablet_copy_service.cc:433] P b92ff49cec804b1f801c11b64e3259b3: ending tablet copy session 13d0f62249b34b01a3a47c6afe10e750-2de731c4261f49088958535b8b7a65e9 on tablet 2de731c4261f49088958535b8b7a65e9 with peer 13d0f62249b34b01a3a47c6afe10e750
I20240503 02:13:43.379716 804 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:43.381542 31049 catalog_manager.cc:5484] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:43.381937 31584 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:43.391139 31452 consensus_queue.cc:237] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:43.391762 2625 tablet_copy_client.cc:523] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:43.392673 804 raft_consensus.cc:1261] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:43.393157 1294 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:43.394620 2625 tablet_bootstrap.cc:492] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap starting.
I20240503 02:13:43.395640 1294 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:43.399202 31036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c425b03a1c64d7ba7a14901d63d7037 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:43.399900 804 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:43.400348 31584 raft_consensus.cc:1261] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:43.400851 31048 catalog_manager.cc:5484] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:43.401849 2501 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:43.404160 31584 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:43.409830 31171 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c425b03a1c64d7ba7a14901d63d7037 with delete_type TABLET_DATA_TOMBSTONED (TS b52c8be37a2241ff9dc555aa0cebd5e5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47568
I20240503 02:13:43.410838 1357 raft_consensus.cc:1050] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 to VOTER
I20240503 02:13:43.411834 1357 consensus_queue.cc:237] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:43.419107 2630 tablet_copy_client.cc:523] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:43.422860 936 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.423156 2630 tablet_bootstrap.cc:492] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap starting.
I20240503 02:13:43.423465 3022 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.430495 31191 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.431227 31584 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.431170 1357 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.431735 3022 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.451351 3033 tablet_replica.cc:330] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5: stopping tablet replica
I20240503 02:13:43.451591 3033 raft_consensus.cc:2229] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:43.451658 1357 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:43.452602 3033 raft_consensus.cc:2258] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:43.453630 3033 ts_tablet_manager.cc:1875] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:43.455580 31583 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:43.466512 31050 catalog_manager.cc:5484] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:43.467464 3033 ts_tablet_manager.cc:1888] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:43.467650 3033 log.cc:1196] T 1c425b03a1c64d7ba7a14901d63d7037 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/1c425b03a1c64d7ba7a14901d63d7037
I20240503 02:13:43.468710 31038 catalog_manager.cc:4830] TS b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): tablet 1c425b03a1c64d7ba7a14901d63d7037 (table rebalance_test_table_4 [id=38897cba33374498b288d375d22641bd]) successfully deleted
I20240503 02:13:43.472323 2625 tablet_bootstrap.cc:492] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:43.473328 2625 tablet_bootstrap.cc:492] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap complete.
I20240503 02:13:43.474242 2625 ts_tablet_manager.cc:1367] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750: Time spent bootstrapping tablet: real 0.080s user 0.044s sys 0.005s
I20240503 02:13:43.475689 2625 raft_consensus.cc:348] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:43.477912 2625 raft_consensus.cc:729] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d0f62249b34b01a3a47c6afe10e750, State: Initialized, Role: LEARNER
I20240503 02:13:43.478469 2625 consensus_queue.cc:260] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:43.479321 936 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:43.480811 31191 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:43.485852 2625 ts_tablet_manager.cc:1398] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:13:43.489348 3039 raft_consensus.cc:1050] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3: attempting to promote NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 to VOTER
I20240503 02:13:43.490375 3039 consensus_queue.cc:237] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:43.495210 31324 consensus_queue.cc:237] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:43.498678 31583 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:43.499411 1357 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:43.501833 3022 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:43.503286 31193 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.504369 3039 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.506445 31583 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:43.510332 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6ce56ca4030a445db890303244caf308 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:43.513177 31050 catalog_manager.cc:5484] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:43.518054 31171 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ce56ca4030a445db890303244caf308 with delete_type TABLET_DATA_TOMBSTONED (TS b52c8be37a2241ff9dc555aa0cebd5e5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47568
I20240503 02:13:43.518436 3033 tablet_replica.cc:330] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5: stopping tablet replica
I20240503 02:13:43.518651 3033 raft_consensus.cc:2229] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:43.519022 3033 raft_consensus.cc:2258] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:43.520007 672 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.520342 3033 ts_tablet_manager.cc:1875] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:43.521112 31324 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.521754 2231 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.522763 2231 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.526580 3033 ts_tablet_manager.cc:1888] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:43.526746 3033 log.cc:1196] T 6ce56ca4030a445db890303244caf308 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/6ce56ca4030a445db890303244caf308
I20240503 02:13:43.527575 31038 catalog_manager.cc:4830] TS b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): tablet 6ce56ca4030a445db890303244caf308 (table rebalance_test_table_6 [id=ad6bdc3b37e64f27824a7eacb47ca790]) successfully deleted
I20240503 02:13:43.534538 935 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:43.535176 3044 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:43.536748 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 13d0f62249b34b01a3a47c6afe10e750-c91a7b003ed9411aba65a46d00c7d95b received from {username='slave'} at 127.23.104.197:33983
I20240503 02:13:43.536880 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 13d0f62249b34b01a3a47c6afe10e750-c91a7b003ed9411aba65a46d00c7d95b on tablet c91a7b003ed9411aba65a46d00c7d95b with peer 13d0f62249b34b01a3a47c6afe10e750
I20240503 02:13:43.538854 936 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:43.542526 3041 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:43.549407 31193 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:43.551401 2630 tablet_bootstrap.cc:492] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:43.551868 2630 tablet_bootstrap.cc:492] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap complete.
I20240503 02:13:43.552193 2630 ts_tablet_manager.cc:1367] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040: Time spent bootstrapping tablet: real 0.129s user 0.063s sys 0.000s
I20240503 02:13:43.552515 31324 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:43.553205 2630 raft_consensus.cc:348] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:43.553484 2630 raft_consensus.cc:729] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0429a57df6ba4f92a73e779b04b05040, State: Initialized, Role: LEARNER
I20240503 02:13:43.553707 2630 consensus_queue.cc:260] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:43.554349 2630 ts_tablet_manager.cc:1398] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:43.555871 31474 tablet_copy_service.cc:342] P fbb7d16e67d94775b2881712419c0577: Request end of tablet copy session 0429a57df6ba4f92a73e779b04b05040-6f8e7774779e47ba9d426ac00ad8be62 received from {username='slave'} at 127.23.104.200:36057
I20240503 02:13:43.556057 31474 tablet_copy_service.cc:433] P fbb7d16e67d94775b2881712419c0577: ending tablet copy session 0429a57df6ba4f92a73e779b04b05040-6f8e7774779e47ba9d426ac00ad8be62 on tablet 6f8e7774779e47ba9d426ac00ad8be62 with peer 0429a57df6ba4f92a73e779b04b05040
I20240503 02:13:43.558373 31050 catalog_manager.cc:5484] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:43.564293 672 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:43.568701 31583 consensus_queue.cc:237] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:43.574012 672 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:43.574630 3041 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:43.576267 31193 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:43.577116 3039 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:43.578042 3039 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:43.580595 31193 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:43.581568 31039 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a5406bce7014bc2a7a1d727cb4aa191 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:43.582799 673 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:43.583500 31050 catalog_manager.cc:5484] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:43.588081 31303 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a5406bce7014bc2a7a1d727cb4aa191 with delete_type TABLET_DATA_TOMBSTONED (TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55596
I20240503 02:13:43.589834 3057 tablet_replica.cc:330] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: stopping tablet replica
I20240503 02:13:43.590086 3057 raft_consensus.cc:2229] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:43.590476 3057 raft_consensus.cc:2258] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:43.592941 3057 ts_tablet_manager.cc:1875] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:43.599035 3057 ts_tablet_manager.cc:1888] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:43.599156 3057 log.cc:1196] T 8a5406bce7014bc2a7a1d727cb4aa191 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/8a5406bce7014bc2a7a1d727cb4aa191
I20240503 02:13:43.599972 31037 catalog_manager.cc:4830] TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): tablet 8a5406bce7014bc2a7a1d727cb4aa191 (table rebalance_test_table_2 [id=88d4dd05cfa84299b1e62ec2a8b6a517]) successfully deleted
I20240503 02:13:43.656885 672 raft_consensus.cc:1203] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:43.718555 1068 raft_consensus.cc:1203] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:43.737532 672 raft_consensus.cc:1203] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:43.748456 672 raft_consensus.cc:1203] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:43.799343 3044 raft_consensus.cc:1050] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 to VOTER
I20240503 02:13:43.799355 3046 raft_consensus.cc:1050] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 0429a57df6ba4f92a73e779b04b05040 to VOTER
I20240503 02:13:43.800263 3044 consensus_queue.cc:237] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:43.800258 3046 consensus_queue.cc:237] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:43.802687 1068 raft_consensus.cc:1261] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.802824 31583 raft_consensus.cc:1261] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.802891 31454 raft_consensus.cc:1261] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.803647 1357 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.804268 31454 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.804237 3045 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.804471 804 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.804917 3045 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.805197 31193 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.805984 3046 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.806252 3044 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.806581 3045 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.809087 3046 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:43.814365 31583 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:43.816154 31048 catalog_manager.cc:5484] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:43.818377 31454 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:43.819219 1068 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:43.823124 31324 consensus_queue.cc:237] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:43.824837 31583 raft_consensus.cc:1261] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:43.825373 3044 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:43.826925 1067 raft_consensus.cc:1261] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:43.827958 3044 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:43.827724 3046 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:43.828362 31583 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:43.831905 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cb7e8fe39c024108be24bc4c144b247b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:43.833251 1067 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:43.836431 3046 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:43.836879 31050 catalog_manager.cc:5484] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:43.838202 31454 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:43.840019 804 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:43.841152 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7e8fe39c024108be24bc4c144b247b with delete_type TABLET_DATA_TOMBSTONED (TS fbb7d16e67d94775b2881712419c0577 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41348
I20240503 02:13:43.844811 31049 catalog_manager.cc:5484] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 reported cstate change: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:43.854311 3157 tablet_replica.cc:330] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577: stopping tablet replica
I20240503 02:13:43.854540 3157 raft_consensus.cc:2229] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:43.854908 3157 raft_consensus.cc:2258] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:43.855700 3157 ts_tablet_manager.cc:1875] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:43.856297 31193 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:43.863476 3157 ts_tablet_manager.cc:1888] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:43.863579 3157 log.cc:1196] T cb7e8fe39c024108be24bc4c144b247b P fbb7d16e67d94775b2881712419c0577: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/cb7e8fe39c024108be24bc4c144b247b
I20240503 02:13:43.864295 31036 catalog_manager.cc:4830] TS fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101): tablet cb7e8fe39c024108be24bc4c144b247b (table rebalance_test_table_7 [id=7a99e03b1a8d47d1a25bbbd747784951]) successfully deleted
I20240503 02:13:43.900020 2501 raft_consensus.cc:1050] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577: attempting to promote NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 to VOTER
I20240503 02:13:43.900864 2501 consensus_queue.cc:237] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:43.903215 31193 raft_consensus.cc:1261] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.903381 31324 raft_consensus.cc:1261] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.903735 3025 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.904259 2963 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.904444 672 raft_consensus.cc:1261] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:43.906116 3025 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:43.907336 1067 raft_consensus.cc:1203] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:43.913460 2963 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:43.916028 31193 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:43.916098 672 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:43.918817 31324 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:43.919608 31044 catalog_manager.cc:5484] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:43.924450 31454 consensus_queue.cc:237] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:43.926522 31324 raft_consensus.cc:1261] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:43.926990 2963 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:43.927944 673 raft_consensus.cc:1261] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:43.929177 3025 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:43.930796 3025 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:43.932607 673 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:43.932708 31324 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:43.934238 31036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c3d978cd00449b8b3edd4c43b657960 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:43.935746 31049 catalog_manager.cc:5484] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:43.939193 31171 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c3d978cd00449b8b3edd4c43b657960 with delete_type TABLET_DATA_TOMBSTONED (TS b52c8be37a2241ff9dc555aa0cebd5e5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47568
I20240503 02:13:43.939538 3033 tablet_replica.cc:330] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5: stopping tablet replica
I20240503 02:13:43.939719 3033 raft_consensus.cc:2229] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:43.939949 3033 raft_consensus.cc:2258] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:43.940726 3033 ts_tablet_manager.cc:1875] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:43.945147 3033 ts_tablet_manager.cc:1888] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:43.945266 3033 log.cc:1196] T 6c3d978cd00449b8b3edd4c43b657960 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/6c3d978cd00449b8b3edd4c43b657960
I20240503 02:13:43.945880 31038 catalog_manager.cc:4830] TS b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): tablet 6c3d978cd00449b8b3edd4c43b657960 (table rebalance_test_table_3 [id=e0a6e74497494a20be21d627357e4eb0]) successfully deleted
I20240503 02:13:44.030519 31324 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c46912d2d4434d4a8c3fa0c3c19ba14b"
dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56324
I20240503 02:13:44.030747 31324 raft_consensus.cc:595] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:44.091630 31324 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c46912d2d4434d4a8c3fa0c3c19ba14b"
dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56344
I20240503 02:13:44.091811 31324 raft_consensus.cc:595] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:44.137312 2179 raft_consensus.cc:1050] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5: attempting to promote NON_VOTER 0429a57df6ba4f92a73e779b04b05040 to VOTER
I20240503 02:13:44.138165 2179 consensus_queue.cc:237] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:44.141543 1067 raft_consensus.cc:1261] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:44.141680 31454 raft_consensus.cc:1261] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:44.141727 31324 raft_consensus.cc:1261] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:44.142205 2179 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:44.142520 3246 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:44.142836 3243 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:44.145987 3246 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:44.146685 31454 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:44.150120 1067 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:44.150645 31324 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:44.151307 31050 catalog_manager.cc:5484] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:44.158741 31193 consensus_queue.cc:237] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:44.160789 31454 raft_consensus.cc:1261] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:44.161075 1067 raft_consensus.cc:1261] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:44.162120 2179 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:44.163333 3040 raft_consensus.cc:1050] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 to VOTER
I20240503 02:13:44.164166 3040 consensus_queue.cc:237] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:44.164816 3244 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:44.165771 3243 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:44.166777 31454 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:44.167066 673 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:44.167587 1357 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:44.168166 1357 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:44.168061 31454 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:44.168674 1067 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:44.170282 31038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4995796bdda64a7cbffa952a05a36673 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:44.173588 31044 catalog_manager.cc:5484] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:44.174741 31193 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:44.175235 3046 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:44.178391 31303 tablet_service.cc:1514] Processing DeleteTablet for tablet 4995796bdda64a7cbffa952a05a36673 with delete_type TABLET_DATA_TOMBSTONED (TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55596
I20240503 02:13:44.181876 3254 tablet_replica.cc:330] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: stopping tablet replica
I20240503 02:13:44.182016 3254 raft_consensus.cc:2229] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:44.182212 3254 raft_consensus.cc:2258] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:44.182922 3254 ts_tablet_manager.cc:1875] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:44.185132 3046 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.185798 673 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.186621 31454 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.189108 3254 ts_tablet_manager.cc:1888] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:44.189282 3254 log.cc:1196] T 4995796bdda64a7cbffa952a05a36673 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/4995796bdda64a7cbffa952a05a36673
I20240503 02:13:44.189813 31191 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.190690 31037 catalog_manager.cc:4830] TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): tablet 4995796bdda64a7cbffa952a05a36673 (table rebalance_test_table_9 [id=3a5935f3f7924b5083fcfc4ee7015893]) successfully deleted
I20240503 02:13:44.191768 31050 catalog_manager.cc:5484] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 reported cstate change: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.196341 31324 consensus_queue.cc:237] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:44.198148 673 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:44.198221 31191 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:44.198702 3046 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:44.199399 3046 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:44.201422 1357 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.203878 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c91a7b003ed9411aba65a46d00c7d95b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:44.204602 31191 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.205516 673 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.205943 31050 catalog_manager.cc:5484] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:44.209579 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet c91a7b003ed9411aba65a46d00c7d95b with delete_type TABLET_DATA_TOMBSTONED (TS fbb7d16e67d94775b2881712419c0577 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41348
I20240503 02:13:44.209915 3157 tablet_replica.cc:330] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577: stopping tablet replica
I20240503 02:13:44.210091 3157 raft_consensus.cc:2229] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:44.210423 3157 raft_consensus.cc:2258] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:44.212010 3157 ts_tablet_manager.cc:1875] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:44.215977 3157 ts_tablet_manager.cc:1888] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:44.216091 3157 log.cc:1196] T c91a7b003ed9411aba65a46d00c7d95b P fbb7d16e67d94775b2881712419c0577: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/c91a7b003ed9411aba65a46d00c7d95b
I20240503 02:13:44.216849 31036 catalog_manager.cc:4830] TS fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101): tablet c91a7b003ed9411aba65a46d00c7d95b (table rebalance_test_table_1 [id=d3a62fd4d46c4ba082abbec8f97c84e0]) successfully deleted
I20240503 02:13:44.217031 3255 raft_consensus.cc:1050] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3: attempting to promote NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 to VOTER
I20240503 02:13:44.217676 3255 consensus_queue.cc:237] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:44.219969 673 raft_consensus.cc:1261] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:44.219986 31454 raft_consensus.cc:1261] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:44.220350 31191 raft_consensus.cc:1261] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:44.220633 3039 consensus_queue.cc:1035] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:44.221184 2231 consensus_queue.cc:1035] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:44.222162 3039 consensus_queue.cc:1035] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:44.228190 3257 raft_consensus.cc:2932] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.228770 31454 raft_consensus.cc:2932] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.229364 673 raft_consensus.cc:2932] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.232367 31191 raft_consensus.cc:2932] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.233530 31044 catalog_manager.cc:5484] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750 reported cstate change: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.238430 31583 consensus_queue.cc:237] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:44.241742 673 raft_consensus.cc:1261] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:44.242080 31191 raft_consensus.cc:1261] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:44.242236 3255 consensus_queue.cc:1035] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:44.242713 2231 consensus_queue.cc:1035] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:44.244791 3039 raft_consensus.cc:2932] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.245609 31191 raft_consensus.cc:2932] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.246677 3040 raft_consensus.cc:979] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: : Instructing follower fbb7d16e67d94775b2881712419c0577 to start an election
I20240503 02:13:44.246809 3040 raft_consensus.cc:1067] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Signalling peer fbb7d16e67d94775b2881712419c0577 to start an election
I20240503 02:13:44.247584 31454 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c46912d2d4434d4a8c3fa0c3c19ba14b"
dest_uuid: "fbb7d16e67d94775b2881712419c0577"
from {username='slave'} at 127.23.104.194:55439
I20240503 02:13:44.247843 31454 raft_consensus.cc:482] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:44.247987 31454 raft_consensus.cc:3037] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:44.247753 673 raft_consensus.cc:2932] T 2de731c4261f49088958535b8b7a65e9 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.250890 31049 catalog_manager.cc:5484] T 2de731c4261f49088958535b8b7a65e9 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index 5 to 6, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.252017 31039 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2de731c4261f49088958535b8b7a65e9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:44.252681 31454 raft_consensus.cc:504] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:44.253822 31324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46912d2d4434d4a8c3fa0c3c19ba14b" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
I20240503 02:13:44.254032 31324 raft_consensus.cc:3032] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:44.254032 31191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46912d2d4434d4a8c3fa0c3c19ba14b" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
I20240503 02:13:44.253991 804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46912d2d4434d4a8c3fa0c3c19ba14b" candidate_uuid: "fbb7d16e67d94775b2881712419c0577" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "05ef0a15decc468ea3a1af8828b0f630"
I20240503 02:13:44.254117 31324 raft_consensus.cc:729] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:44.254184 31191 raft_consensus.cc:3037] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:44.254314 804 raft_consensus.cc:3037] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:44.254384 31324 consensus_queue.cc:260] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:44.254696 31324 raft_consensus.cc:3037] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:44.256112 31454 leader_election.cc:288] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 2 election: Requested vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195)
I20240503 02:13:44.256624 31324 raft_consensus.cc:2454] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 2.
I20240503 02:13:44.256708 31191 raft_consensus.cc:2454] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 2.
I20240503 02:13:44.257361 31388 leader_election.cc:302] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: b52c8be37a2241ff9dc555aa0cebd5e5, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, fbb7d16e67d94775b2881712419c0577; no voters:
I20240503 02:13:44.257674 2963 raft_consensus.cc:2781] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:44.257826 2963 raft_consensus.cc:686] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 2 LEADER]: Becoming Leader. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Running, Role: LEADER
I20240503 02:13:44.258128 2963 consensus_queue.cc:237] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:44.258932 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet 2de731c4261f49088958535b8b7a65e9 with delete_type TABLET_DATA_TOMBSTONED (TS fbb7d16e67d94775b2881712419c0577 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41348
I20240503 02:13:44.259258 3157 tablet_replica.cc:330] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577: stopping tablet replica
I20240503 02:13:44.259449 3157 raft_consensus.cc:2229] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:44.259697 3157 raft_consensus.cc:2258] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:44.261157 3157 ts_tablet_manager.cc:1875] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:44.261281 804 raft_consensus.cc:2454] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb7d16e67d94775b2881712419c0577 in term 2.
I20240503 02:13:44.261888 31048 catalog_manager.cc:5484] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 reported cstate change: term changed from 1 to 2, leader changed from cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) to fbb7d16e67d94775b2881712419c0577 (127.23.104.195). New cstate: current_term: 2 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:44.265715 3157 ts_tablet_manager.cc:1888] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:44.265838 3157 log.cc:1196] T 2de731c4261f49088958535b8b7a65e9 P fbb7d16e67d94775b2881712419c0577: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/2de731c4261f49088958535b8b7a65e9
I20240503 02:13:44.266461 31036 catalog_manager.cc:4830] TS fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101): tablet 2de731c4261f49088958535b8b7a65e9 (table rebalance_test_table_7 [id=7a99e03b1a8d47d1a25bbbd747784951]) successfully deleted
I20240503 02:13:44.270401 3255 raft_consensus.cc:1050] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3: attempting to promote NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 to VOTER
I20240503 02:13:44.270992 3255 consensus_queue.cc:237] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:44.272868 31454 raft_consensus.cc:1261] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:44.272866 673 raft_consensus.cc:1261] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:44.272871 31191 raft_consensus.cc:1261] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:44.273324 3039 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:44.273607 3255 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:44.273825 3257 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:44.276999 3257 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.277649 673 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.278360 31191 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.282660 31454 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.283339 31048 catalog_manager.cc:5484] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750 reported cstate change: config changed from index 4 to 5, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.288794 31583 consensus_queue.cc:237] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:44.290230 673 raft_consensus.cc:1261] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:44.290328 31454 raft_consensus.cc:1261] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:44.290776 3265 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:44.291118 2231 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:44.292984 3257 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.293673 31454 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.293798 673 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:44.296252 31039 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8d8d17ee831446ae8d4a0417ede894b8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:44.296988 31050 catalog_manager.cc:5484] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 5 to 6, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:44.300571 31171 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8d17ee831446ae8d4a0417ede894b8 with delete_type TABLET_DATA_TOMBSTONED (TS b52c8be37a2241ff9dc555aa0cebd5e5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47568
I20240503 02:13:44.300922 3033 tablet_replica.cc:330] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5: stopping tablet replica
I20240503 02:13:44.301121 3033 raft_consensus.cc:2229] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:44.301386 3033 raft_consensus.cc:2258] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:44.302101 3033 ts_tablet_manager.cc:1875] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:44.306807 3033 ts_tablet_manager.cc:1888] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:44.306921 3033 log.cc:1196] T 8d8d17ee831446ae8d4a0417ede894b8 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/8d8d17ee831446ae8d4a0417ede894b8
I20240503 02:13:44.307591 31038 catalog_manager.cc:4830] TS b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): tablet 8d8d17ee831446ae8d4a0417ede894b8 (table rebalance_test_table_2 [id=88d4dd05cfa84299b1e62ec2a8b6a517]) successfully deleted
I20240503 02:13:44.412830 2963 raft_consensus.cc:1050] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577: attempting to promote NON_VOTER 0429a57df6ba4f92a73e779b04b05040 to VOTER
I20240503 02:13:44.413645 2963 consensus_queue.cc:237] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:44.415800 31324 raft_consensus.cc:1261] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:44.415748 1067 raft_consensus.cc:1261] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:44.416165 31191 raft_consensus.cc:1261] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:44.416304 3159 consensus_queue.cc:1035] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:44.416833 3159 consensus_queue.cc:1035] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:44.417047 2963 consensus_queue.cc:1035] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:44.429953 3159 raft_consensus.cc:2932] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:44.433490 1067 raft_consensus.cc:2932] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:44.436419 31048 catalog_manager.cc:5484] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:44.439746 31324 raft_consensus.cc:2932] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:44.440948 31454 consensus_queue.cc:237] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:44.439543 31191 raft_consensus.cc:2932] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:44.445765 1067 raft_consensus.cc:1261] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:44.446290 2963 consensus_queue.cc:1035] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:44.448662 2501 raft_consensus.cc:2932] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:44.449409 31193 raft_consensus.cc:1261] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:44.450075 3159 consensus_queue.cc:1035] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:44.450337 1068 raft_consensus.cc:2932] T 6f8e7774779e47ba9d426ac00ad8be62 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:44.451771 31193 raft_consensus.cc:2932] T 6f8e7774779e47ba9d426ac00ad8be62 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:44.453455 31036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6f8e7774779e47ba9d426ac00ad8be62 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:44.453943 31050 catalog_manager.cc:5484] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 5 to 6, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:44.457422 31303 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8e7774779e47ba9d426ac00ad8be62 with delete_type TABLET_DATA_TOMBSTONED (TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55596
I20240503 02:13:44.457783 3254 tablet_replica.cc:330] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: stopping tablet replica
I20240503 02:13:44.457978 3254 raft_consensus.cc:2229] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:44.458204 3254 raft_consensus.cc:2258] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:44.459019 3254 ts_tablet_manager.cc:1875] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:44.463281 3254 ts_tablet_manager.cc:1888] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:44.463377 3254 log.cc:1196] T 6f8e7774779e47ba9d426ac00ad8be62 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f8e7774779e47ba9d426ac00ad8be62
I20240503 02:13:44.464079 31037 catalog_manager.cc:4830] TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): tablet 6f8e7774779e47ba9d426ac00ad8be62 (table rebalance_test_table_5 [id=f64903d295f2485983d7a34ee454f2b9]) successfully deleted
I20240503 02:13:44.655617 804 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 2 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:44.656214 3159 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:44.725162 31193 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:44.725620 3159 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:44.729475 31324 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:44.730118 2963 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:44.733075 31454 consensus_queue.cc:237] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:44.735141 31193 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:44.735637 2501 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:44.735705 804 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 2 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:44.736367 2501 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:44.737957 2963 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:44.738498 31193 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:44.739948 804 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:44.741775 31036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c46912d2d4434d4a8c3fa0c3c19ba14b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:44.744660 31048 catalog_manager.cc:5484] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index 5 to 7, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New cstate: current_term: 2 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:44.748129 31303 tablet_service.cc:1514] Processing DeleteTablet for tablet c46912d2d4434d4a8c3fa0c3c19ba14b with delete_type TABLET_DATA_TOMBSTONED (TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55596
I20240503 02:13:44.748474 3254 tablet_replica.cc:330] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: stopping tablet replica
I20240503 02:13:44.748682 3254 raft_consensus.cc:2229] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:44.748865 3254 raft_consensus.cc:2258] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:44.749706 3254 ts_tablet_manager.cc:1875] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:44.754156 3254 ts_tablet_manager.cc:1888] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:13:44.754253 3254 log.cc:1196] T c46912d2d4434d4a8c3fa0c3c19ba14b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/c46912d2d4434d4a8c3fa0c3c19ba14b
I20240503 02:13:44.754766 31037 catalog_manager.cc:4830] TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): tablet c46912d2d4434d4a8c3fa0c3c19ba14b (table rebalance_test_table_7 [id=7a99e03b1a8d47d1a25bbbd747784951]) successfully deleted
I20240503 02:13:44.953814 1048 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:45.037402 652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:45.045287 31563 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:45.046710 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:45.051921 31433 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:45.058357 31303 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:13:45.061753 31171 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:13:45.062351 1951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:45.087070 784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:45.220625 31324 consensus_queue.cc:237] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:45.223160 1068 raft_consensus.cc:1261] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.223863 31584 raft_consensus.cc:1261] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.224727 3046 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.225134 3040 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.231693 3040 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:45.232221 1068 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
W20240503 02:13:45.235991 31257 consensus_peers.cc:485] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601): Couldn't send request to peer 6ad20578b4a74bfabb6d1404de034a65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b76fa26b48e942ed840b5ee55ce40387. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:45.242050 31584 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:45.242959 31050 catalog_manager.cc:5484] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 reported cstate change: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:45.257016 31193 consensus_queue.cc:237] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:45.260326 1068 raft_consensus.cc:1261] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.260421 31454 raft_consensus.cc:1261] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.261153 2179 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.261695 3403 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.264474 3403 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:45.268062 31454 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
W20240503 02:13:45.270085 31127 consensus_peers.cc:485] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 -> Peer 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997): Couldn't send request to peer 4a3e3937b61447598fd76bc57f5d0939. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4995796bdda64a7cbffa952a05a36673. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:45.272964 1068 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:45.274394 31050 catalog_manager.cc:5484] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:45.289031 31324 consensus_queue.cc:237] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:45.291162 31193 raft_consensus.cc:1261] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:45.291502 31260 consensus_peers.cc:485] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401): Couldn't send request to peer 0429a57df6ba4f92a73e779b04b05040. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f8353133ccd4601a3761652546a33b5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:45.291529 935 raft_consensus.cc:1261] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.291689 3395 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.292079 3046 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.294988 3046 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:45.295576 31193 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:45.295620 935 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:45.300423 31044 catalog_manager.cc:5484] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 6 to 7, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:45.319980 31584 consensus_queue.cc:237] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:45.323041 31454 raft_consensus.cc:1261] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.323179 673 raft_consensus.cc:1261] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.323737 3257 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.324134 3423 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:13:45.330945 31519 consensus_peers.cc:485] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 -> Peer 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195): Couldn't send request to peer 05ef0a15decc468ea3a1af8828b0f630. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d8d17ee831446ae8d4a0417ede894b8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:45.333276 3423 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:45.333999 673 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:45.334714 31454 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:45.338752 31050 catalog_manager.cc:5484] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 6 to 7, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:45.352411 31584 consensus_queue.cc:237] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:45.354403 31324 raft_consensus.cc:1261] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.354439 673 raft_consensus.cc:1261] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:45.354552 31518 consensus_peers.cc:485] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 -> Peer 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997): Couldn't send request to peer 4a3e3937b61447598fd76bc57f5d0939. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80a15c3541fc4068acac595bd909f8d8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:45.354916 3257 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.355170 3423 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.357692 3257 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:45.358310 31324 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:45.362250 673 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:45.364058 31050 catalog_manager.cc:5484] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:45.375936 31324 consensus_queue.cc:237] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:45.378115 31584 raft_consensus.cc:1261] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.378377 804 raft_consensus.cc:1261] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:45.378371 31257 consensus_peers.cc:485] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601): Couldn't send request to peer 6ad20578b4a74bfabb6d1404de034a65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e964b80f5414d9ea3a5aea07b99d1d4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:45.378871 3040 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.379148 3046 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.381683 3395 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:45.382266 31584 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:45.382450 804 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:45.386145 31050 catalog_manager.cc:5484] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:45.400905 31324 consensus_queue.cc:237] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
W20240503 02:13:45.403453 31258 consensus_peers.cc:485] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543): Couldn't send request to peer 13d0f62249b34b01a3a47c6afe10e750. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c681bdcf3074c54b1d533cf1f7d2b75. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:45.403497 935 raft_consensus.cc:1261] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.403533 31454 raft_consensus.cc:1261] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.404182 3040 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.406810 3040 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:45.409840 31454 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:45.412081 3395 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240503 02:13:45.414805 31044 catalog_manager.cc:5484] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 6 to 7, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:45.415048 935 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } } }
I20240503 02:13:45.438016 31454 consensus_queue.cc:237] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:45.440521 804 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 2 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:13:45.441533 2501 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:45.444103 31193 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:13:45.444275 2501 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:45.448007 804 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
W20240503 02:13:45.449682 31388 consensus_peers.cc:485] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 -> Peer 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997): Couldn't send request to peer 4a3e3937b61447598fd76bc57f5d0939. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c46912d2d4434d4a8c3fa0c3c19ba14b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:45.452170 3459 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:45.454015 31044 catalog_manager.cc:5484] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 reported cstate change: config changed from index 7 to 8, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New cstate: current_term: 2 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:45.454967 31193 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:45.466564 31324 consensus_queue.cc:237] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:45.468813 31193 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.468833 673 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:45.469010 31260 consensus_peers.cc:485] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401): Couldn't send request to peer 0429a57df6ba4f92a73e779b04b05040. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c91a7b003ed9411aba65a46d00c7d95b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:45.469368 3391 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.469627 3395 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.472133 3395 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:45.472913 673 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:45.476040 31193 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:45.478252 31044 catalog_manager.cc:5484] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 reported cstate change: config changed from index 6 to 7, NON_VOTER 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } } }
I20240503 02:13:45.491672 31324 consensus_queue.cc:237] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:45.493726 935 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.493921 31584 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:45.494143 31259 consensus_peers.cc:485] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195): Couldn't send request to peer 05ef0a15decc468ea3a1af8828b0f630. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ce56ca4030a445db890303244caf308. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:45.494273 3040 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.494515 3391 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.496980 3395 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:45.497602 31584 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:45.497701 935 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:45.502560 31046 catalog_manager.cc:5484] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 6 to 7, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:45.520383 31324 consensus_queue.cc:237] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:45.522293 804 raft_consensus.cc:1261] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:45.522389 31257 consensus_peers.cc:485] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601): Couldn't send request to peer 6ad20578b4a74bfabb6d1404de034a65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7626c06dc67448f0b65f67b401d62c88. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:45.522413 31584 raft_consensus.cc:1261] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.522850 3040 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.523121 3395 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.525564 3040 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:45.526141 31584 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:45.528729 804 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:45.530936 31044 catalog_manager.cc:5484] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:45.548265 31454 consensus_queue.cc:237] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:45.550730 31584 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.550743 673 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:45.550812 31388 consensus_peers.cc:485] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 -> Peer 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997): Couldn't send request to peer 4a3e3937b61447598fd76bc57f5d0939. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9000ce76eb154434a88b52e357497767. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:45.551214 3459 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.551483 2501 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.553575 3459 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:45.554209 673 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:45.570843 31046 catalog_manager.cc:5484] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 reported cstate change: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:45.585618 31584 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:45.601792 31324 consensus_queue.cc:237] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:45.604532 31193 raft_consensus.cc:1261] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.605124 673 raft_consensus.cc:1261] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:45.605089 31257 consensus_peers.cc:485] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601): Couldn't send request to peer 6ad20578b4a74bfabb6d1404de034a65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b909a66859144f7b4b649d505c9645e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:45.605616 3395 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.605978 3040 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.610796 3391 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:45.616665 31193 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:45.621896 31048 catalog_manager.cc:5484] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:45.637609 673 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:45.661696 31454 consensus_queue.cc:237] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
W20240503 02:13:45.664544 31389 consensus_peers.cc:485] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 -> Peer 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195): Couldn't send request to peer 05ef0a15decc468ea3a1af8828b0f630. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c3d978cd00449b8b3edd4c43b657960. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:45.664920 31324 raft_consensus.cc:1261] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.665035 673 raft_consensus.cc:1261] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.665410 3461 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.665732 2501 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.668637 3459 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:45.669164 31324 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:45.669384 673 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:45.675154 31048 catalog_manager.cc:5484] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 6 to 7, NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) added. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } } }
I20240503 02:13:45.696494 31454 consensus_queue.cc:237] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:45.699275 804 raft_consensus.cc:1261] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.699852 31584 raft_consensus.cc:1261] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:45.699883 3511 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:45.700470 3459 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:13:45.706596 31387 consensus_peers.cc:485] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 -> Peer 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601): Couldn't send request to peer 6ad20578b4a74bfabb6d1404de034a65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7230f1d648c34351aed29f98795150a2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:45.722405 3517 ts_tablet_manager.cc:902] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939: Initiating tablet copy from peer b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:45.733512 3521 ts_tablet_manager.cc:902] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:45.737264 3511 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:45.740705 803 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:45.745818 31048 catalog_manager.cc:5484] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:45.773711 3527 ts_tablet_manager.cc:902] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:45.775636 3531 ts_tablet_manager.cc:902] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630: Initiating tablet copy from peer b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:45.776716 31584 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:45.784276 3540 ts_tablet_manager.cc:902] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939: Initiating tablet copy from peer b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:45.793790 3540 tablet_copy_client.cc:308] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:39849
I20240503 02:13:45.794715 31604 tablet_copy_service.cc:140] P b92ff49cec804b1f801c11b64e3259b3: Received BeginTabletCopySession request for tablet 80a15c3541fc4068acac595bd909f8d8 from peer 4a3e3937b61447598fd76bc57f5d0939 ({username='slave'} at 127.23.104.199:40937)
I20240503 02:13:45.794879 31604 tablet_copy_service.cc:161] P b92ff49cec804b1f801c11b64e3259b3: Beginning new tablet copy session on tablet 80a15c3541fc4068acac595bd909f8d8 from peer 4a3e3937b61447598fd76bc57f5d0939 at {username='slave'} at 127.23.104.199:40937: session id = 4a3e3937b61447598fd76bc57f5d0939-80a15c3541fc4068acac595bd909f8d8
I20240503 02:13:45.800681 31604 tablet_copy_source_session.cc:205] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:45.800915 3531 tablet_copy_client.cc:308] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:39849
I20240503 02:13:45.802831 3517 tablet_copy_client.cc:308] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.193:44019
I20240503 02:13:45.807045 3540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80a15c3541fc4068acac595bd909f8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:45.831885 31604 tablet_copy_service.cc:140] P b92ff49cec804b1f801c11b64e3259b3: Received BeginTabletCopySession request for tablet 8d8d17ee831446ae8d4a0417ede894b8 from peer 05ef0a15decc468ea3a1af8828b0f630 ({username='slave'} at 127.23.104.198:54891)
I20240503 02:13:45.832103 31604 tablet_copy_service.cc:161] P b92ff49cec804b1f801c11b64e3259b3: Beginning new tablet copy session on tablet 8d8d17ee831446ae8d4a0417ede894b8 from peer 05ef0a15decc468ea3a1af8828b0f630 at {username='slave'} at 127.23.104.198:54891: session id = 05ef0a15decc468ea3a1af8828b0f630-8d8d17ee831446ae8d4a0417ede894b8
I20240503 02:13:45.836417 31604 tablet_copy_source_session.cc:205] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:45.838173 3531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8d17ee831446ae8d4a0417ede894b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:45.847983 3540 tablet_copy_client.cc:783] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:45.848390 3540 tablet_copy_client.cc:647] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:45.853993 3531 tablet_copy_client.cc:783] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:45.854836 3531 tablet_copy_client.cc:647] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:45.855120 3545 tablet_copy_client.cc:1002] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:45.857098 3547 tablet_copy_client.cc:1002] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:45.862558 31213 tablet_copy_service.cc:140] P b52c8be37a2241ff9dc555aa0cebd5e5: Received BeginTabletCopySession request for tablet 4995796bdda64a7cbffa952a05a36673 from peer 4a3e3937b61447598fd76bc57f5d0939 ({username='slave'} at 127.23.104.199:33733)
I20240503 02:13:45.862737 31213 tablet_copy_service.cc:161] P b52c8be37a2241ff9dc555aa0cebd5e5: Beginning new tablet copy session on tablet 4995796bdda64a7cbffa952a05a36673 from peer 4a3e3937b61447598fd76bc57f5d0939 at {username='slave'} at 127.23.104.199:33733: session id = 4a3e3937b61447598fd76bc57f5d0939-4995796bdda64a7cbffa952a05a36673
I20240503 02:13:45.865885 31213 tablet_copy_source_session.cc:205] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:45.867290 3517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4995796bdda64a7cbffa952a05a36673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:45.875015 3517 tablet_copy_client.cc:783] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:45.875247 3517 tablet_copy_client.cc:647] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:45.876145 3521 tablet_copy_client.cc:308] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:45.877794 3537 tablet_copy_client.cc:1002] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:45.891516 3527 tablet_copy_client.cc:308] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:45.892273 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet 3f8353133ccd4601a3761652546a33b5 from peer 0429a57df6ba4f92a73e779b04b05040 ({username='slave'} at 127.23.104.200:44199)
I20240503 02:13:45.892423 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet 3f8353133ccd4601a3761652546a33b5 from peer 0429a57df6ba4f92a73e779b04b05040 at {username='slave'} at 127.23.104.200:44199: session id = 0429a57df6ba4f92a73e779b04b05040-3f8353133ccd4601a3761652546a33b5
I20240503 02:13:45.896440 31344 tablet_copy_source_session.cc:205] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:45.897939 3527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8353133ccd4601a3761652546a33b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:45.901269 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet b76fa26b48e942ed840b5ee55ce40387 from peer 6ad20578b4a74bfabb6d1404de034a65 ({username='slave'} at 127.23.104.201:50129)
I20240503 02:13:45.901473 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet b76fa26b48e942ed840b5ee55ce40387 from peer 6ad20578b4a74bfabb6d1404de034a65 at {username='slave'} at 127.23.104.201:50129: session id = 6ad20578b4a74bfabb6d1404de034a65-b76fa26b48e942ed840b5ee55ce40387
I20240503 02:13:45.906695 31344 tablet_copy_source_session.cc:205] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:45.913115 3527 tablet_copy_client.cc:783] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:45.913425 3527 tablet_copy_client.cc:647] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:45.918634 3521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76fa26b48e942ed840b5ee55ce40387. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:45.923978 3555 tablet_copy_client.cc:1002] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:45.936617 3521 tablet_copy_client.cc:783] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:45.937036 3521 tablet_copy_client.cc:647] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:45.939006 3552 tablet_copy_client.cc:1002] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:45.975225 3581 ts_tablet_manager.cc:902] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:45.984467 3584 ts_tablet_manager.cc:902] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:45.985137 3585 ts_tablet_manager.cc:902] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939: Initiating tablet copy from peer fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:45.990859 3584 tablet_copy_client.cc:308] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:45.992580 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet 9e964b80f5414d9ea3a5aea07b99d1d4 from peer 6ad20578b4a74bfabb6d1404de034a65 ({username='slave'} at 127.23.104.201:50129)
I20240503 02:13:45.992766 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet 9e964b80f5414d9ea3a5aea07b99d1d4 from peer 6ad20578b4a74bfabb6d1404de034a65 at {username='slave'} at 127.23.104.201:50129: session id = 6ad20578b4a74bfabb6d1404de034a65-9e964b80f5414d9ea3a5aea07b99d1d4
I20240503 02:13:45.996374 31344 tablet_copy_source_session.cc:205] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:45.997853 3584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e964b80f5414d9ea3a5aea07b99d1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:46.005331 3581 tablet_copy_client.cc:308] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:46.005949 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet 8c681bdcf3074c54b1d533cf1f7d2b75 from peer 13d0f62249b34b01a3a47c6afe10e750 ({username='slave'} at 127.23.104.197:33983)
I20240503 02:13:46.006115 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet 8c681bdcf3074c54b1d533cf1f7d2b75 from peer 13d0f62249b34b01a3a47c6afe10e750 at {username='slave'} at 127.23.104.197:33983: session id = 13d0f62249b34b01a3a47c6afe10e750-8c681bdcf3074c54b1d533cf1f7d2b75
I20240503 02:13:46.006381 3584 tablet_copy_client.cc:783] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:46.006604 3584 tablet_copy_client.cc:647] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:46.009027 3590 tablet_copy_client.cc:1002] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:46.012578 31344 tablet_copy_source_session.cc:205] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:46.013957 3581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c681bdcf3074c54b1d533cf1f7d2b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:46.023111 3581 tablet_copy_client.cc:783] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:46.023394 3581 tablet_copy_client.cc:647] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:46.026490 3595 tablet_copy_client.cc:1002] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:46.027662 3585 tablet_copy_client.cc:308] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:39101
I20240503 02:13:46.037025 31474 tablet_copy_service.cc:140] P fbb7d16e67d94775b2881712419c0577: Received BeginTabletCopySession request for tablet 9000ce76eb154434a88b52e357497767 from peer 4a3e3937b61447598fd76bc57f5d0939 ({username='slave'} at 127.23.104.199:55707)
I20240503 02:13:46.037178 31474 tablet_copy_service.cc:161] P fbb7d16e67d94775b2881712419c0577: Beginning new tablet copy session on tablet 9000ce76eb154434a88b52e357497767 from peer 4a3e3937b61447598fd76bc57f5d0939 at {username='slave'} at 127.23.104.199:55707: session id = 4a3e3937b61447598fd76bc57f5d0939-9000ce76eb154434a88b52e357497767
I20240503 02:13:46.039979 31474 tablet_copy_source_session.cc:205] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:46.041424 3585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9000ce76eb154434a88b52e357497767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:46.042174 3598 ts_tablet_manager.cc:902] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939: Initiating tablet copy from peer fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:46.045730 3603 ts_tablet_manager.cc:902] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630: Initiating tablet copy from peer fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:46.051450 3585 tablet_copy_client.cc:783] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:46.051709 3585 tablet_copy_client.cc:647] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:46.054695 3597 tablet_copy_client.cc:1002] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:46.060412 3603 tablet_copy_client.cc:308] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:39101
I20240503 02:13:46.061403 31474 tablet_copy_service.cc:140] P fbb7d16e67d94775b2881712419c0577: Received BeginTabletCopySession request for tablet 6c3d978cd00449b8b3edd4c43b657960 from peer 05ef0a15decc468ea3a1af8828b0f630 ({username='slave'} at 127.23.104.198:53527)
I20240503 02:13:46.061604 31474 tablet_copy_service.cc:161] P fbb7d16e67d94775b2881712419c0577: Beginning new tablet copy session on tablet 6c3d978cd00449b8b3edd4c43b657960 from peer 05ef0a15decc468ea3a1af8828b0f630 at {username='slave'} at 127.23.104.198:53527: session id = 05ef0a15decc468ea3a1af8828b0f630-6c3d978cd00449b8b3edd4c43b657960
I20240503 02:13:46.065496 31474 tablet_copy_source_session.cc:205] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:46.068152 3598 tablet_copy_client.cc:308] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:39101
I20240503 02:13:46.069046 31473 tablet_copy_service.cc:140] P fbb7d16e67d94775b2881712419c0577: Received BeginTabletCopySession request for tablet c46912d2d4434d4a8c3fa0c3c19ba14b from peer 4a3e3937b61447598fd76bc57f5d0939 ({username='slave'} at 127.23.104.199:55707)
I20240503 02:13:46.069219 31473 tablet_copy_service.cc:161] P fbb7d16e67d94775b2881712419c0577: Beginning new tablet copy session on tablet c46912d2d4434d4a8c3fa0c3c19ba14b from peer 4a3e3937b61447598fd76bc57f5d0939 at {username='slave'} at 127.23.104.199:55707: session id = 4a3e3937b61447598fd76bc57f5d0939-c46912d2d4434d4a8c3fa0c3c19ba14b
I20240503 02:13:46.069993 3613 ts_tablet_manager.cc:902] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:46.071035 3603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3d978cd00449b8b3edd4c43b657960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:46.073210 31473 tablet_copy_source_session.cc:205] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:46.074424 3598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46912d2d4434d4a8c3fa0c3c19ba14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:46.077260 3603 tablet_copy_client.cc:783] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:46.077600 3603 tablet_copy_client.cc:647] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:46.079298 3611 tablet_copy_client.cc:1002] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:46.080682 3598 tablet_copy_client.cc:783] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:46.082013 3613 tablet_copy_client.cc:308] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:46.082183 3598 tablet_copy_client.cc:647] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:46.082620 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet c91a7b003ed9411aba65a46d00c7d95b from peer 0429a57df6ba4f92a73e779b04b05040 ({username='slave'} at 127.23.104.200:44199)
I20240503 02:13:46.082767 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet c91a7b003ed9411aba65a46d00c7d95b from peer 0429a57df6ba4f92a73e779b04b05040 at {username='slave'} at 127.23.104.200:44199: session id = 0429a57df6ba4f92a73e779b04b05040-c91a7b003ed9411aba65a46d00c7d95b
I20240503 02:13:46.083917 3607 tablet_copy_client.cc:1002] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:46.085101 31344 tablet_copy_source_session.cc:205] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:46.086139 3613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91a7b003ed9411aba65a46d00c7d95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:46.091451 3613 tablet_copy_client.cc:783] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:46.091670 3613 tablet_copy_client.cc:647] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:46.092784 3622 ts_tablet_manager.cc:902] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:46.093343 3619 tablet_copy_client.cc:1002] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:46.094449 3622 tablet_copy_client.cc:308] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:46.096401 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet 6ce56ca4030a445db890303244caf308 from peer 05ef0a15decc468ea3a1af8828b0f630 ({username='slave'} at 127.23.104.198:42437)
I20240503 02:13:46.096544 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet 6ce56ca4030a445db890303244caf308 from peer 05ef0a15decc468ea3a1af8828b0f630 at {username='slave'} at 127.23.104.198:42437: session id = 05ef0a15decc468ea3a1af8828b0f630-6ce56ca4030a445db890303244caf308
I20240503 02:13:46.098572 31344 tablet_copy_source_session.cc:205] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:46.099709 3622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ce56ca4030a445db890303244caf308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:46.106478 3622 tablet_copy_client.cc:783] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:46.106794 3622 tablet_copy_client.cc:647] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:46.108119 3627 tablet_copy_client.cc:1002] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:46.134364 3645 ts_tablet_manager.cc:902] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65: Initiating tablet copy from peer fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:46.144668 3645 tablet_copy_client.cc:308] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:39101
I20240503 02:13:46.145367 3648 ts_tablet_manager.cc:902] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:46.152287 31474 tablet_copy_service.cc:140] P fbb7d16e67d94775b2881712419c0577: Received BeginTabletCopySession request for tablet 7230f1d648c34351aed29f98795150a2 from peer 6ad20578b4a74bfabb6d1404de034a65 ({username='slave'} at 127.23.104.201:38237)
I20240503 02:13:46.152459 31474 tablet_copy_service.cc:161] P fbb7d16e67d94775b2881712419c0577: Beginning new tablet copy session on tablet 7230f1d648c34351aed29f98795150a2 from peer 6ad20578b4a74bfabb6d1404de034a65 at {username='slave'} at 127.23.104.201:38237: session id = 6ad20578b4a74bfabb6d1404de034a65-7230f1d648c34351aed29f98795150a2
I20240503 02:13:46.155097 3648 tablet_copy_client.cc:308] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:46.156158 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet 7626c06dc67448f0b65f67b401d62c88 from peer 6ad20578b4a74bfabb6d1404de034a65 ({username='slave'} at 127.23.104.201:50129)
I20240503 02:13:46.156297 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet 7626c06dc67448f0b65f67b401d62c88 from peer 6ad20578b4a74bfabb6d1404de034a65 at {username='slave'} at 127.23.104.201:50129: session id = 6ad20578b4a74bfabb6d1404de034a65-7626c06dc67448f0b65f67b401d62c88
I20240503 02:13:46.157757 31474 tablet_copy_source_session.cc:205] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:46.159425 31344 tablet_copy_source_session.cc:205] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:46.160605 3645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7230f1d648c34351aed29f98795150a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:46.166673 3648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7626c06dc67448f0b65f67b401d62c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:46.169154 3645 tablet_copy_client.cc:783] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:46.169414 3645 tablet_copy_client.cc:647] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:46.171567 3655 tablet_copy_client.cc:1002] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:46.172547 3660 ts_tablet_manager.cc:902] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:46.175122 3648 tablet_copy_client.cc:783] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:46.175355 3648 tablet_copy_client.cc:647] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:46.177068 3659 tablet_copy_client.cc:1002] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:46.179464 3660 tablet_copy_client.cc:308] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:46.180075 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet 9b909a66859144f7b4b649d505c9645e from peer 6ad20578b4a74bfabb6d1404de034a65 ({username='slave'} at 127.23.104.201:50129)
I20240503 02:13:46.180215 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet 9b909a66859144f7b4b649d505c9645e from peer 6ad20578b4a74bfabb6d1404de034a65 at {username='slave'} at 127.23.104.201:50129: session id = 6ad20578b4a74bfabb6d1404de034a65-9b909a66859144f7b4b649d505c9645e
I20240503 02:13:46.182600 31344 tablet_copy_source_session.cc:205] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:46.183753 3660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b909a66859144f7b4b649d505c9645e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:46.189760 3660 tablet_copy_client.cc:783] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:46.189988 3660 tablet_copy_client.cc:647] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:46.191371 3669 tablet_copy_client.cc:1002] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:47.355495 3540 tablet_copy_client.cc:523] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.357617 3531 tablet_copy_client.cc:523] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.360077 3540 tablet_bootstrap.cc:492] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap starting.
I20240503 02:13:47.360186 3531 tablet_bootstrap.cc:492] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap starting.
I20240503 02:13:47.378109 3517 tablet_copy_client.cc:523] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.381671 3517 tablet_bootstrap.cc:492] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap starting.
I20240503 02:13:47.413658 3531 tablet_bootstrap.cc:492] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:47.414219 3531 tablet_bootstrap.cc:492] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap complete.
I20240503 02:13:47.414568 3531 ts_tablet_manager.cc:1367] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630: Time spent bootstrapping tablet: real 0.054s user 0.041s sys 0.004s
I20240503 02:13:47.415437 3531 raft_consensus.cc:348] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:47.415843 3531 raft_consensus.cc:729] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05ef0a15decc468ea3a1af8828b0f630, State: Initialized, Role: LEARNER
I20240503 02:13:47.416163 3531 consensus_queue.cc:260] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:47.424366 3527 tablet_copy_client.cc:523] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.427762 3527 tablet_bootstrap.cc:492] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap starting.
I20240503 02:13:47.429188 3531 ts_tablet_manager.cc:1398] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20240503 02:13:47.430424 3540 tablet_bootstrap.cc:492] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:47.430588 31604 tablet_copy_service.cc:342] P b92ff49cec804b1f801c11b64e3259b3: Request end of tablet copy session 05ef0a15decc468ea3a1af8828b0f630-8d8d17ee831446ae8d4a0417ede894b8 received from {username='slave'} at 127.23.104.198:54891
I20240503 02:13:47.430714 31604 tablet_copy_service.cc:433] P b92ff49cec804b1f801c11b64e3259b3: ending tablet copy session 05ef0a15decc468ea3a1af8828b0f630-8d8d17ee831446ae8d4a0417ede894b8 on tablet 8d8d17ee831446ae8d4a0417ede894b8 with peer 05ef0a15decc468ea3a1af8828b0f630
I20240503 02:13:47.430858 3540 tablet_bootstrap.cc:492] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap complete.
I20240503 02:13:47.431097 3540 ts_tablet_manager.cc:1367] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939: Time spent bootstrapping tablet: real 0.071s user 0.053s sys 0.009s
I20240503 02:13:47.432000 3540 raft_consensus.cc:348] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:47.432325 3540 raft_consensus.cc:729] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a3e3937b61447598fd76bc57f5d0939, State: Initialized, Role: LEARNER
I20240503 02:13:47.432581 3540 consensus_queue.cc:260] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:47.433465 3540 ts_tablet_manager.cc:1398] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:47.435740 31604 tablet_copy_service.cc:342] P b92ff49cec804b1f801c11b64e3259b3: Request end of tablet copy session 4a3e3937b61447598fd76bc57f5d0939-80a15c3541fc4068acac595bd909f8d8 received from {username='slave'} at 127.23.104.199:40937
I20240503 02:13:47.435878 31604 tablet_copy_service.cc:433] P b92ff49cec804b1f801c11b64e3259b3: ending tablet copy session 4a3e3937b61447598fd76bc57f5d0939-80a15c3541fc4068acac595bd909f8d8 on tablet 80a15c3541fc4068acac595bd909f8d8 with peer 4a3e3937b61447598fd76bc57f5d0939
I20240503 02:13:47.439525 3521 tablet_copy_client.cc:523] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.444025 3521 tablet_bootstrap.cc:492] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap starting.
I20240503 02:13:47.456964 3521 log.cc:824] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:47.459494 3517 tablet_bootstrap.cc:492] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:47.460148 3517 tablet_bootstrap.cc:492] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap complete.
I20240503 02:13:47.460549 3517 ts_tablet_manager.cc:1367] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939: Time spent bootstrapping tablet: real 0.079s user 0.055s sys 0.013s
I20240503 02:13:47.461727 3517 raft_consensus.cc:348] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:47.462344 3517 raft_consensus.cc:729] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a3e3937b61447598fd76bc57f5d0939, State: Initialized, Role: LEARNER
I20240503 02:13:47.462663 3517 consensus_queue.cc:260] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:47.463357 3517 ts_tablet_manager.cc:1398] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:47.464994 31213 tablet_copy_service.cc:342] P b52c8be37a2241ff9dc555aa0cebd5e5: Request end of tablet copy session 4a3e3937b61447598fd76bc57f5d0939-4995796bdda64a7cbffa952a05a36673 received from {username='slave'} at 127.23.104.199:33733
I20240503 02:13:47.465130 31213 tablet_copy_service.cc:433] P b52c8be37a2241ff9dc555aa0cebd5e5: ending tablet copy session 4a3e3937b61447598fd76bc57f5d0939-4995796bdda64a7cbffa952a05a36673 on tablet 4995796bdda64a7cbffa952a05a36673 with peer 4a3e3937b61447598fd76bc57f5d0939
I20240503 02:13:47.482182 3527 tablet_bootstrap.cc:492] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:47.482637 3527 tablet_bootstrap.cc:492] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040: Bootstrap complete.
I20240503 02:13:47.482923 3527 ts_tablet_manager.cc:1367] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040: Time spent bootstrapping tablet: real 0.055s user 0.049s sys 0.001s
I20240503 02:13:47.483595 3527 raft_consensus.cc:348] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:47.483816 3527 raft_consensus.cc:729] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0429a57df6ba4f92a73e779b04b05040, State: Initialized, Role: LEARNER
I20240503 02:13:47.484014 3527 consensus_queue.cc:260] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:47.484547 3527 ts_tablet_manager.cc:1398] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:47.485397 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 0429a57df6ba4f92a73e779b04b05040-3f8353133ccd4601a3761652546a33b5 received from {username='slave'} at 127.23.104.200:44199
I20240503 02:13:47.485524 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 0429a57df6ba4f92a73e779b04b05040-3f8353133ccd4601a3761652546a33b5 on tablet 3f8353133ccd4601a3761652546a33b5 with peer 0429a57df6ba4f92a73e779b04b05040
I20240503 02:13:47.509480 3584 tablet_copy_client.cc:523] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.512642 3521 tablet_bootstrap.cc:492] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:47.513073 3584 tablet_bootstrap.cc:492] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap starting.
I20240503 02:13:47.513149 3521 tablet_bootstrap.cc:492] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap complete.
I20240503 02:13:47.513602 3521 ts_tablet_manager.cc:1367] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65: Time spent bootstrapping tablet: real 0.070s user 0.063s sys 0.003s
I20240503 02:13:47.520768 3521 raft_consensus.cc:348] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:47.521317 3521 raft_consensus.cc:729] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ad20578b4a74bfabb6d1404de034a65, State: Initialized, Role: LEARNER
I20240503 02:13:47.521914 3521 consensus_queue.cc:260] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:47.523901 3521 ts_tablet_manager.cc:1398] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20240503 02:13:47.524825 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 6ad20578b4a74bfabb6d1404de034a65-b76fa26b48e942ed840b5ee55ce40387 received from {username='slave'} at 127.23.104.201:50129
I20240503 02:13:47.524953 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 6ad20578b4a74bfabb6d1404de034a65-b76fa26b48e942ed840b5ee55ce40387 on tablet b76fa26b48e942ed840b5ee55ce40387 with peer 6ad20578b4a74bfabb6d1404de034a65
I20240503 02:13:47.526866 3581 tablet_copy_client.cc:523] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.531648 3581 tablet_bootstrap.cc:492] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap starting.
I20240503 02:13:47.555083 3585 tablet_copy_client.cc:523] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.558583 3585 tablet_bootstrap.cc:492] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap starting.
I20240503 02:13:47.569706 3584 tablet_bootstrap.cc:492] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:47.570293 3584 tablet_bootstrap.cc:492] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap complete.
I20240503 02:13:47.570638 3584 ts_tablet_manager.cc:1367] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65: Time spent bootstrapping tablet: real 0.058s user 0.047s sys 0.005s
I20240503 02:13:47.571719 3584 raft_consensus.cc:348] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:47.572111 3584 raft_consensus.cc:729] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ad20578b4a74bfabb6d1404de034a65, State: Initialized, Role: LEARNER
I20240503 02:13:47.572382 3584 consensus_queue.cc:260] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:47.573155 3584 ts_tablet_manager.cc:1398] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:47.573827 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 6ad20578b4a74bfabb6d1404de034a65-9e964b80f5414d9ea3a5aea07b99d1d4 received from {username='slave'} at 127.23.104.201:50129
I20240503 02:13:47.573966 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 6ad20578b4a74bfabb6d1404de034a65-9e964b80f5414d9ea3a5aea07b99d1d4 on tablet 9e964b80f5414d9ea3a5aea07b99d1d4 with peer 6ad20578b4a74bfabb6d1404de034a65
I20240503 02:13:47.579701 3603 tablet_copy_client.cc:523] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.583638 3603 tablet_bootstrap.cc:492] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap starting.
I20240503 02:13:47.584254 3598 tablet_copy_client.cc:523] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.589138 3598 tablet_bootstrap.cc:492] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap starting.
I20240503 02:13:47.593775 3613 tablet_copy_client.cc:523] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.599056 3613 tablet_bootstrap.cc:492] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040: Bootstrap starting.
I20240503 02:13:47.608914 3622 tablet_copy_client.cc:523] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.615314 3622 tablet_bootstrap.cc:492] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap starting.
I20240503 02:13:47.622464 1972 raft_consensus.cc:1203] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:47.646441 1068 raft_consensus.cc:1203] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:47.666035 3585 tablet_bootstrap.cc:492] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:47.670454 3603 tablet_bootstrap.cc:492] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:47.671159 3603 tablet_bootstrap.cc:492] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap complete.
I20240503 02:13:47.671504 3603 ts_tablet_manager.cc:1367] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630: Time spent bootstrapping tablet: real 0.088s user 0.057s sys 0.002s
I20240503 02:13:47.673134 3645 tablet_copy_client.cc:523] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.677403 3648 tablet_copy_client.cc:523] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.678653 3645 tablet_bootstrap.cc:492] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap starting.
I20240503 02:13:47.681380 3648 tablet_bootstrap.cc:492] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap starting.
I20240503 02:13:47.683004 3585 tablet_bootstrap.cc:492] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap complete.
I20240503 02:13:47.684075 3585 ts_tablet_manager.cc:1367] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939: Time spent bootstrapping tablet: real 0.126s user 0.056s sys 0.000s
I20240503 02:13:47.685619 3585 raft_consensus.cc:348] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:47.687656 3585 raft_consensus.cc:729] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a3e3937b61447598fd76bc57f5d0939, State: Initialized, Role: LEARNER
I20240503 02:13:47.688015 3585 consensus_queue.cc:260] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:47.690815 3585 ts_tablet_manager.cc:1398] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:47.693284 3660 tablet_copy_client.cc:523] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.696393 31474 tablet_copy_service.cc:342] P fbb7d16e67d94775b2881712419c0577: Request end of tablet copy session 4a3e3937b61447598fd76bc57f5d0939-9000ce76eb154434a88b52e357497767 received from {username='slave'} at 127.23.104.199:55707
I20240503 02:13:47.696511 31474 tablet_copy_service.cc:433] P fbb7d16e67d94775b2881712419c0577: ending tablet copy session 4a3e3937b61447598fd76bc57f5d0939-9000ce76eb154434a88b52e357497767 on tablet 9000ce76eb154434a88b52e357497767 with peer 4a3e3937b61447598fd76bc57f5d0939
I20240503 02:13:47.697989 935 raft_consensus.cc:1203] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:47.701508 3660 tablet_bootstrap.cc:492] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap starting.
I20240503 02:13:47.712805 3603 raft_consensus.cc:348] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:47.713121 3603 raft_consensus.cc:729] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05ef0a15decc468ea3a1af8828b0f630, State: Initialized, Role: LEARNER
I20240503 02:13:47.713346 3603 consensus_queue.cc:260] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:47.713960 3603 ts_tablet_manager.cc:1398] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:47.721863 31474 tablet_copy_service.cc:342] P fbb7d16e67d94775b2881712419c0577: Request end of tablet copy session 05ef0a15decc468ea3a1af8828b0f630-6c3d978cd00449b8b3edd4c43b657960 received from {username='slave'} at 127.23.104.198:53527
I20240503 02:13:47.722012 31474 tablet_copy_service.cc:433] P fbb7d16e67d94775b2881712419c0577: ending tablet copy session 05ef0a15decc468ea3a1af8828b0f630-6c3d978cd00449b8b3edd4c43b657960 on tablet 6c3d978cd00449b8b3edd4c43b657960 with peer 05ef0a15decc468ea3a1af8828b0f630
I20240503 02:13:47.733551 3581 tablet_bootstrap.cc:492] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240503 02:13:47.734022 31518 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:37543, user_credentials={real_user=slave}} blocked reactor thread for 61363.4us
I20240503 02:13:47.734752 935 raft_consensus.cc:1203] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:47.739141 3581 tablet_bootstrap.cc:492] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750: Bootstrap complete.
I20240503 02:13:47.741251 3581 ts_tablet_manager.cc:1367] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750: Time spent bootstrapping tablet: real 0.210s user 0.064s sys 0.009s
I20240503 02:13:47.744529 3581 raft_consensus.cc:348] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:47.745004 3581 raft_consensus.cc:729] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d0f62249b34b01a3a47c6afe10e750, State: Initialized, Role: LEARNER
I20240503 02:13:47.746969 3581 consensus_queue.cc:260] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: true } }
I20240503 02:13:47.760397 3581 ts_tablet_manager.cc:1398] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240503 02:13:47.761925 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 13d0f62249b34b01a3a47c6afe10e750-8c681bdcf3074c54b1d533cf1f7d2b75 received from {username='slave'} at 127.23.104.197:33983
I20240503 02:13:47.762046 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 13d0f62249b34b01a3a47c6afe10e750-8c681bdcf3074c54b1d533cf1f7d2b75 on tablet 8c681bdcf3074c54b1d533cf1f7d2b75 with peer 13d0f62249b34b01a3a47c6afe10e750
I20240503 02:13:47.772716 804 raft_consensus.cc:1203] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:47.786816 3622 tablet_bootstrap.cc:492] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:47.787288 3622 tablet_bootstrap.cc:492] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630: Bootstrap complete.
I20240503 02:13:47.787624 3622 ts_tablet_manager.cc:1367] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630: Time spent bootstrapping tablet: real 0.173s user 0.059s sys 0.006s
I20240503 02:13:47.788373 3622 raft_consensus.cc:348] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:47.788579 3622 raft_consensus.cc:729] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05ef0a15decc468ea3a1af8828b0f630, State: Initialized, Role: LEARNER
I20240503 02:13:47.788790 3622 consensus_queue.cc:260] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: NON_VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: true } }
I20240503 02:13:47.789484 3622 ts_tablet_manager.cc:1398] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:47.790362 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 05ef0a15decc468ea3a1af8828b0f630-6ce56ca4030a445db890303244caf308 received from {username='slave'} at 127.23.104.198:42437
I20240503 02:13:47.790482 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 05ef0a15decc468ea3a1af8828b0f630-6ce56ca4030a445db890303244caf308 on tablet 6ce56ca4030a445db890303244caf308 with peer 05ef0a15decc468ea3a1af8828b0f630
I20240503 02:13:47.795094 3613 tablet_bootstrap.cc:492] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:47.803818 3613 tablet_bootstrap.cc:492] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040: Bootstrap complete.
I20240503 02:13:47.804169 3613 ts_tablet_manager.cc:1367] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040: Time spent bootstrapping tablet: real 0.205s user 0.042s sys 0.015s
I20240503 02:13:47.808043 3613 raft_consensus.cc:348] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:47.808440 3613 raft_consensus.cc:729] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0429a57df6ba4f92a73e779b04b05040, State: Initialized, Role: LEARNER
I20240503 02:13:47.808713 3613 consensus_queue.cc:260] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: true } }
I20240503 02:13:47.809257 3613 ts_tablet_manager.cc:1398] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:13:47.813330 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 0429a57df6ba4f92a73e779b04b05040-c91a7b003ed9411aba65a46d00c7d95b received from {username='slave'} at 127.23.104.200:44199
I20240503 02:13:47.813463 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 0429a57df6ba4f92a73e779b04b05040-c91a7b003ed9411aba65a46d00c7d95b on tablet c91a7b003ed9411aba65a46d00c7d95b with peer 0429a57df6ba4f92a73e779b04b05040
I20240503 02:13:47.823812 3648 tablet_bootstrap.cc:492] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:47.824499 3648 tablet_bootstrap.cc:492] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap complete.
I20240503 02:13:47.824900 3648 ts_tablet_manager.cc:1367] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65: Time spent bootstrapping tablet: real 0.144s user 0.047s sys 0.008s
I20240503 02:13:47.825990 3648 raft_consensus.cc:348] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:47.826509 3648 raft_consensus.cc:729] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ad20578b4a74bfabb6d1404de034a65, State: Initialized, Role: LEARNER
I20240503 02:13:47.826869 3648 consensus_queue.cc:260] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:47.827849 3648 ts_tablet_manager.cc:1398] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:47.828567 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 6ad20578b4a74bfabb6d1404de034a65-7626c06dc67448f0b65f67b401d62c88 received from {username='slave'} at 127.23.104.201:50129
I20240503 02:13:47.828727 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 6ad20578b4a74bfabb6d1404de034a65-7626c06dc67448f0b65f67b401d62c88 on tablet 7626c06dc67448f0b65f67b401d62c88 with peer 6ad20578b4a74bfabb6d1404de034a65
I20240503 02:13:47.829237 3598 tablet_bootstrap.cc:492] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:47.829694 3598 tablet_bootstrap.cc:492] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap complete.
I20240503 02:13:47.830042 3598 ts_tablet_manager.cc:1367] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939: Time spent bootstrapping tablet: real 0.241s user 0.047s sys 0.016s
I20240503 02:13:47.830224 3660 tablet_bootstrap.cc:492] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:47.830705 3660 tablet_bootstrap.cc:492] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap complete.
I20240503 02:13:47.830904 3598 raft_consensus.cc:348] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:47.831187 3598 raft_consensus.cc:729] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4a3e3937b61447598fd76bc57f5d0939, State: Initialized, Role: LEARNER
I20240503 02:13:47.831396 3598 consensus_queue.cc:260] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:47.832039 3598 ts_tablet_manager.cc:1398] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:47.832516 3660 ts_tablet_manager.cc:1367] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65: Time spent bootstrapping tablet: real 0.131s user 0.049s sys 0.007s
I20240503 02:13:47.834542 31474 tablet_copy_service.cc:342] P fbb7d16e67d94775b2881712419c0577: Request end of tablet copy session 4a3e3937b61447598fd76bc57f5d0939-c46912d2d4434d4a8c3fa0c3c19ba14b received from {username='slave'} at 127.23.104.199:55707
I20240503 02:13:47.834653 31474 tablet_copy_service.cc:433] P fbb7d16e67d94775b2881712419c0577: ending tablet copy session 4a3e3937b61447598fd76bc57f5d0939-c46912d2d4434d4a8c3fa0c3c19ba14b on tablet c46912d2d4434d4a8c3fa0c3c19ba14b with peer 4a3e3937b61447598fd76bc57f5d0939
I20240503 02:13:47.835731 3660 raft_consensus.cc:348] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:47.836078 3660 raft_consensus.cc:729] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ad20578b4a74bfabb6d1404de034a65, State: Initialized, Role: LEARNER
I20240503 02:13:47.836334 3660 consensus_queue.cc:260] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:47.838543 3660 ts_tablet_manager.cc:1398] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:13:47.839386 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 6ad20578b4a74bfabb6d1404de034a65-9b909a66859144f7b4b649d505c9645e received from {username='slave'} at 127.23.104.201:50129
I20240503 02:13:47.839511 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 6ad20578b4a74bfabb6d1404de034a65-9b909a66859144f7b4b649d505c9645e on tablet 9b909a66859144f7b4b649d505c9645e with peer 6ad20578b4a74bfabb6d1404de034a65
I20240503 02:13:47.846916 3645 tablet_bootstrap.cc:492] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:47.847509 3645 tablet_bootstrap.cc:492] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap complete.
I20240503 02:13:47.847822 3645 ts_tablet_manager.cc:1367] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65: Time spent bootstrapping tablet: real 0.169s user 0.050s sys 0.005s
I20240503 02:13:47.849944 3645 raft_consensus.cc:348] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:47.850208 3645 raft_consensus.cc:729] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ad20578b4a74bfabb6d1404de034a65, State: Initialized, Role: LEARNER
I20240503 02:13:47.850420 3645 consensus_queue.cc:260] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:47.850991 3645 ts_tablet_manager.cc:1398] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:13:47.851982 31474 tablet_copy_service.cc:342] P fbb7d16e67d94775b2881712419c0577: Request end of tablet copy session 6ad20578b4a74bfabb6d1404de034a65-7230f1d648c34351aed29f98795150a2 received from {username='slave'} at 127.23.104.201:38237
I20240503 02:13:47.852120 31474 tablet_copy_service.cc:433] P fbb7d16e67d94775b2881712419c0577: ending tablet copy session 6ad20578b4a74bfabb6d1404de034a65-7230f1d648c34351aed29f98795150a2 on tablet 7230f1d648c34351aed29f98795150a2 with peer 6ad20578b4a74bfabb6d1404de034a65
I20240503 02:13:47.900427 804 raft_consensus.cc:1203] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:47.964800 1972 raft_consensus.cc:1203] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:47.974012 935 raft_consensus.cc:1203] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:47.984441 1972 raft_consensus.cc:1203] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:48.006848 1068 raft_consensus.cc:1203] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:48.018205 935 raft_consensus.cc:1203] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 02:13:48.075933 3511 raft_consensus.cc:1050] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577: attempting to promote NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 to VOTER
I20240503 02:13:48.076803 3511 consensus_queue.cc:237] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:48.079157 935 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.079259 673 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.079301 31584 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.079769 4006 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.080085 3511 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.080348 4005 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.083163 3511 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.083770 31584 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.085788 935 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.088107 31046 catalog_manager.cc:5484] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.089625 673 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.123266 673 raft_consensus.cc:1203] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:48.125907 3950 raft_consensus.cc:1050] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3: attempting to promote NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 to VOTER
I20240503 02:13:48.126732 3950 consensus_queue.cc:237] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:48.130303 673 raft_consensus.cc:1261] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.130437 936 raft_consensus.cc:1261] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.130580 31323 raft_consensus.cc:1261] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.131067 3675 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.131893 3675 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.132668 3957 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.135937 3957 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.136588 673 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.136929 31323 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.139357 936 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.140661 31048 catalog_manager.cc:5484] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:48.146723 31584 consensus_queue.cc:237] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:48.148736 3391 raft_consensus.cc:1050] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 0429a57df6ba4f92a73e779b04b05040 to VOTER
I20240503 02:13:48.148738 673 raft_consensus.cc:1261] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.149621 3391 consensus_queue.cc:237] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:48.149914 3950 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.154361 3950 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.158538 31039 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80a15c3541fc4068acac595bd909f8d8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:48.159411 3046 raft_consensus.cc:1050] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 to VOTER
I20240503 02:13:48.160034 673 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.160449 3046 consensus_queue.cc:237] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:48.161918 936 raft_consensus.cc:1261] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.163834 3675 consensus_queue.cc:1035] T 80a15c3541fc4068acac595bd909f8d8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.166661 1068 raft_consensus.cc:1261] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.167694 936 raft_consensus.cc:2932] T 80a15c3541fc4068acac595bd909f8d8 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.169080 31044 catalog_manager.cc:5484] T 80a15c3541fc4068acac595bd909f8d8 P 13d0f62249b34b01a3a47c6afe10e750 reported cstate change: config changed from index 8 to 9, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.171415 935 raft_consensus.cc:1261] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.172389 31303 tablet_service.cc:1514] Processing DeleteTablet for tablet 80a15c3541fc4068acac595bd909f8d8 with delete_type TABLET_DATA_TOMBSTONED (TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55596
I20240503 02:13:48.173290 4025 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.178746 4029 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.179443 1972 raft_consensus.cc:1261] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.179613 1068 raft_consensus.cc:1261] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.180194 4033 tablet_replica.cc:330] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: stopping tablet replica
I20240503 02:13:48.180860 4033 raft_consensus.cc:2229] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:48.180884 4027 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.181284 31191 raft_consensus.cc:1261] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.181725 4027 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.181733 4033 raft_consensus.cc:2258] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:48.182148 4027 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.184468 4033 ts_tablet_manager.cc:1875] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:48.206131 4033 ts_tablet_manager.cc:1888] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:48.207115 4033 log.cc:1196] T 80a15c3541fc4068acac595bd909f8d8 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/80a15c3541fc4068acac595bd909f8d8
I20240503 02:13:48.207469 3675 raft_consensus.cc:1050] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3: attempting to promote NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 to VOTER
I20240503 02:13:48.210279 31037 catalog_manager.cc:4830] TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): tablet 80a15c3541fc4068acac595bd909f8d8 (table rebalance_test_table_0 [id=cc3c62c772ec4610892f67e8c6d52379]) successfully deleted
I20240503 02:13:48.210043 4025 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:48.211455 3675 consensus_queue.cc:237] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:48.214629 1068 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:48.215077 31193 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:48.220479 31044 catalog_manager.cc:5484] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 7 to 8, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:48.222729 1972 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.226327 31584 raft_consensus.cc:1261] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.231191 3046 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.236372 31584 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
W20240503 02:13:48.237706 31258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:36997, user_credentials={real_user=slave}} blocked reactor thread for 65747us
I20240503 02:13:48.213078 4030 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.246268 804 raft_consensus.cc:1261] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.246598 673 raft_consensus.cc:1261] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.247351 3957 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.247725 31046 catalog_manager.cc:5484] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.249125 31454 raft_consensus.cc:1261] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.250433 3950 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.252220 3957 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.253350 1068 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.253949 936 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:48.261065 31323 consensus_queue.cc:237] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:48.268429 1067 raft_consensus.cc:1261] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.269450 4031 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.271768 1972 raft_consensus.cc:1261] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.272614 3046 consensus_queue.cc:1035] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.273571 3046 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.276721 1067 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.282518 3675 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.286074 804 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.287765 31044 catalog_manager.cc:5484] T b76fa26b48e942ed840b5ee55ce40387 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:48.289744 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b76fa26b48e942ed840b5ee55ce40387 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:48.290181 1972 raft_consensus.cc:2932] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.291656 31563 tablet_service.cc:1514] Processing DeleteTablet for tablet b76fa26b48e942ed840b5ee55ce40387 with delete_type TABLET_DATA_TOMBSTONED (TS b92ff49cec804b1f801c11b64e3259b3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36462
I20240503 02:13:48.293231 31044 catalog_manager.cc:5484] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630 reported cstate change: config changed from index 7 to 8, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.295729 671 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.297171 31454 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.303323 4049 tablet_replica.cc:330] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3: stopping tablet replica
I20240503 02:13:48.303594 4049 raft_consensus.cc:2229] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:48.305362 4049 raft_consensus.cc:2258] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:48.306105 31584 consensus_queue.cc:237] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:48.307363 4049 ts_tablet_manager.cc:1875] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:48.308884 804 raft_consensus.cc:1261] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.309149 1972 raft_consensus.cc:1203] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:48.309556 3950 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.311558 31453 raft_consensus.cc:1261] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.312920 3957 consensus_queue.cc:1035] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.313977 3957 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.315510 4049 ts_tablet_manager.cc:1888] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:48.315680 4049 log.cc:1196] T b76fa26b48e942ed840b5ee55ce40387 P b92ff49cec804b1f801c11b64e3259b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/b76fa26b48e942ed840b5ee55ce40387
I20240503 02:13:48.316671 804 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.317179 31039 catalog_manager.cc:4830] TS b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849): tablet b76fa26b48e942ed840b5ee55ce40387 (table rebalance_test_table_2 [id=88d4dd05cfa84299b1e62ec2a8b6a517]) successfully deleted
I20240503 02:13:48.317555 31454 raft_consensus.cc:2932] T 8d8d17ee831446ae8d4a0417ede894b8 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.319634 31039 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8d8d17ee831446ae8d4a0417ede894b8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:48.321152 1972 raft_consensus.cc:1203] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:48.322868 31048 catalog_manager.cc:5484] T 8d8d17ee831446ae8d4a0417ede894b8 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 8 to 9, VOTER 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) evicted. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:48.346418 804 raft_consensus.cc:1203] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:48.358611 31454 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9000ce76eb154434a88b52e357497767"
dest_uuid: "fbb7d16e67d94775b2881712419c0577"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38544
I20240503 02:13:48.358834 31454 raft_consensus.cc:595] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:48.372071 652 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8d17ee831446ae8d4a0417ede894b8 with delete_type TABLET_DATA_TOMBSTONED (TS 13d0f62249b34b01a3a47c6afe10e750 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43020
I20240503 02:13:48.372929 4078 tablet_replica.cc:330] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750: stopping tablet replica
I20240503 02:13:48.373395 4078 raft_consensus.cc:2229] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:48.373880 4078 raft_consensus.cc:2258] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:48.375324 4078 ts_tablet_manager.cc:1875] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:48.381218 4078 ts_tablet_manager.cc:1888] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:48.381374 4078 log.cc:1196] T 8d8d17ee831446ae8d4a0417ede894b8 P 13d0f62249b34b01a3a47c6afe10e750: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/wals/8d8d17ee831446ae8d4a0417ede894b8
I20240503 02:13:48.382145 31037 catalog_manager.cc:4830] TS 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543): tablet 8d8d17ee831446ae8d4a0417ede894b8 (table rebalance_test_table_2 [id=88d4dd05cfa84299b1e62ec2a8b6a517]) successfully deleted
I20240503 02:13:48.391176 3046 raft_consensus.cc:1050] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 to VOTER
I20240503 02:13:48.392033 3046 consensus_queue.cc:237] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:48.396674 31584 raft_consensus.cc:1261] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.397511 804 raft_consensus.cc:1261] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.397858 4032 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.398751 1972 raft_consensus.cc:1261] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.399067 4032 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.399547 4032 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.404412 804 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.403362 4032 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.414896 31048 catalog_manager.cc:5484] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 reported cstate change: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.415380 1971 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.419307 4006 raft_consensus.cc:1050] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577: attempting to promote NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 to VOTER
I20240503 02:13:48.420279 4006 consensus_queue.cc:237] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:48.423305 31323 consensus_queue.cc:237] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:48.424343 31584 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.426060 804 raft_consensus.cc:1261] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.426698 4034 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.428918 1970 raft_consensus.cc:1261] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.430856 3046 consensus_queue.cc:1035] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.434080 4034 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.439941 31193 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:48.440732 4005 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.442318 936 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939 [term 2 LEARNER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:48.443986 803 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 2 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:48.444635 3511 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.445173 3511 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.447125 4098 raft_consensus.cc:1050] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5: attempting to promote NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 to VOTER
I20240503 02:13:48.448526 4098 consensus_queue.cc:237] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:48.459570 1970 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.466699 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9e964b80f5414d9ea3a5aea07b99d1d4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:48.469761 31044 catalog_manager.cc:5484] T 9e964b80f5414d9ea3a5aea07b99d1d4 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:48.476630 31563 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e964b80f5414d9ea3a5aea07b99d1d4 with delete_type TABLET_DATA_TOMBSTONED (TS b92ff49cec804b1f801c11b64e3259b3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36462
I20240503 02:13:48.477085 4049 tablet_replica.cc:330] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3: stopping tablet replica
I20240503 02:13:48.477348 4049 raft_consensus.cc:2229] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:48.477753 4049 raft_consensus.cc:2258] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:48.478726 4049 ts_tablet_manager.cc:1875] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:48.486027 4049 ts_tablet_manager.cc:1888] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:48.486233 4049 log.cc:1196] T 9e964b80f5414d9ea3a5aea07b99d1d4 P b92ff49cec804b1f801c11b64e3259b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/9e964b80f5414d9ea3a5aea07b99d1d4
I20240503 02:13:48.487053 31039 catalog_manager.cc:4830] TS b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849): tablet 9e964b80f5414d9ea3a5aea07b99d1d4 (table rebalance_test_table_1 [id=d3a62fd4d46c4ba082abbec8f97c84e0]) successfully deleted
I20240503 02:13:48.488593 802 raft_consensus.cc:2932] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.502008 4005 raft_consensus.cc:1050] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577: attempting to promote NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 to VOTER
I20240503 02:13:48.503171 4005 consensus_queue.cc:237] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:48.507354 4031 raft_consensus.cc:1050] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 0429a57df6ba4f92a73e779b04b05040 to VOTER
I20240503 02:13:48.508277 4031 consensus_queue.cc:237] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:48.512944 31584 raft_consensus.cc:1261] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.513763 3511 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.517241 671 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.518332 1067 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.519459 4030 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.520354 4030 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.526793 671 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:48.531970 802 raft_consensus.cc:1261] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.533674 3511 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.536254 1970 raft_consensus.cc:1261] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.538615 4005 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.539824 4034 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:48.541687 3511 raft_consensus.cc:1050] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577: attempting to promote NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 to VOTER
I20240503 02:13:48.542418 3511 consensus_queue.cc:237] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:48.549100 4005 raft_consensus.cc:979] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577: : Instructing follower b92ff49cec804b1f801c11b64e3259b3 to start an election
I20240503 02:13:48.549278 4005 raft_consensus.cc:1067] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Signalling peer b92ff49cec804b1f801c11b64e3259b3 to start an election
I20240503 02:13:48.550040 31583 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9000ce76eb154434a88b52e357497767"
dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
from {username='slave'} at 127.23.104.195:40533
I20240503 02:13:48.550256 31583 raft_consensus.cc:482] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:48.550343 31583 raft_consensus.cc:3037] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:48.553156 31583 raft_consensus.cc:504] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:48.554421 673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9000ce76eb154434a88b52e357497767" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "13d0f62249b34b01a3a47c6afe10e750"
I20240503 02:13:48.554711 673 raft_consensus.cc:3037] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:48.557585 673 raft_consensus.cc:2454] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 2.
I20240503 02:13:48.559044 31454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9000ce76eb154434a88b52e357497767" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "fbb7d16e67d94775b2881712419c0577"
I20240503 02:13:48.560204 935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9000ce76eb154434a88b52e357497767" candidate_uuid: "b92ff49cec804b1f801c11b64e3259b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4a3e3937b61447598fd76bc57f5d0939"
I20240503 02:13:48.560477 935 raft_consensus.cc:3037] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:48.564067 31583 leader_election.cc:288] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 2 election: Requested vote from peers fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101), 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543), 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997)
I20240503 02:13:48.565361 935 raft_consensus.cc:2454] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 2.
I20240503 02:13:48.566128 31518 leader_election.cc:302] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 13d0f62249b34b01a3a47c6afe10e750, 4a3e3937b61447598fd76bc57f5d0939, b92ff49cec804b1f801c11b64e3259b3; no voters:
I20240503 02:13:48.566533 3953 raft_consensus.cc:2781] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:48.566707 3953 raft_consensus.cc:686] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 2 LEADER]: Becoming Leader. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Running, Role: LEADER
I20240503 02:13:48.567124 3953 consensus_queue.cc:237] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:48.580551 31454 raft_consensus.cc:3032] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:48.580734 31454 raft_consensus.cc:729] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fbb7d16e67d94775b2881712419c0577, State: Running, Role: LEADER
I20240503 02:13:48.580624 31044 catalog_manager.cc:5484] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 reported cstate change: config changed from index 7 to 8, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:48.581074 31454 consensus_queue.cc:260] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:48.581552 31454 raft_consensus.cc:3037] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:48.584666 31454 raft_consensus.cc:2454] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92ff49cec804b1f801c11b64e3259b3 in term 2.
I20240503 02:13:48.589464 804 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.593888 4005 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.605896 31044 catalog_manager.cc:5484] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630 reported cstate change: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.618510 1972 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.639650 31454 consensus_queue.cc:237] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:48.644214 1972 raft_consensus.cc:1261] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.644840 804 raft_consensus.cc:1261] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.644790 4096 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.645449 4115 consensus_queue.cc:1035] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.647929 4113 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.648499 1970 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.648742 801 raft_consensus.cc:2932] T 7230f1d648c34351aed29f98795150a2 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.652112 31036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7230f1d648c34351aed29f98795150a2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:48.653920 31048 catalog_manager.cc:5484] T 7230f1d648c34351aed29f98795150a2 P 6ad20578b4a74bfabb6d1404de034a65 reported cstate change: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.656134 4005 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.658053 31563 tablet_service.cc:1514] Processing DeleteTablet for tablet 7230f1d648c34351aed29f98795150a2 with delete_type TABLET_DATA_TOMBSTONED (TS b92ff49cec804b1f801c11b64e3259b3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36462
I20240503 02:13:48.660706 4049 tablet_replica.cc:330] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3: stopping tablet replica
I20240503 02:13:48.660916 4049 raft_consensus.cc:2229] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:48.661099 4049 pending_rounds.cc:62] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3: Trying to abort 1 pending ops.
I20240503 02:13:48.661204 4049 pending_rounds.cc:68] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023421147146113024 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "7230f1d648c34351aed29f98795150a2" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } } }
I20240503 02:13:48.661568 4049 raft_consensus.cc:2894] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240503 02:13:48.661661 4049 raft_consensus.cc:2866] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:13:48.661761 4049 raft_consensus.cc:2258] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:48.662586 4049 ts_tablet_manager.cc:1875] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:48.664031 31044 catalog_manager.cc:5484] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: term changed from 1 to 2, leader changed from fbb7d16e67d94775b2881712419c0577 (127.23.104.195) to b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196). New cstate: current_term: 2 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:48.668252 936 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.669726 4049 ts_tablet_manager.cc:1888] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:48.669898 4049 log.cc:1196] T 7230f1d648c34351aed29f98795150a2 P b92ff49cec804b1f801c11b64e3259b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/7230f1d648c34351aed29f98795150a2
I20240503 02:13:48.670800 31039 catalog_manager.cc:4830] TS b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849): tablet 7230f1d648c34351aed29f98795150a2 (table rebalance_test_table_8 [id=dc947496e186485092d5d621503a4e6f]) successfully deleted
I20240503 02:13:48.673964 31048 catalog_manager.cc:5484] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 8 to 9, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:48.676839 803 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.679394 31454 consensus_queue.cc:237] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:48.681491 31193 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.681885 936 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:48.682535 4032 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.682713 4117 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:48.684834 31191 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:48.685487 4096 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.688716 31036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c46912d2d4434d4a8c3fa0c3c19ba14b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:48.689486 804 raft_consensus.cc:1261] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.690182 936 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.690810 1067 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:48.690522 31044 catalog_manager.cc:5484] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 9 to 10, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New cstate: current_term: 2 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:48.695251 31171 tablet_service.cc:1514] Processing DeleteTablet for tablet c46912d2d4434d4a8c3fa0c3c19ba14b with delete_type TABLET_DATA_TOMBSTONED (TS b52c8be37a2241ff9dc555aa0cebd5e5 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:47568
I20240503 02:13:48.696437 4096 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.700256 801 raft_consensus.cc:1261] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 2 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:48.703320 31323 raft_consensus.cc:1261] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.709089 936 raft_consensus.cc:1261] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.709636 4096 consensus_queue.cc:1035] T c46912d2d4434d4a8c3fa0c3c19ba14b P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:48.709805 4094 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.709993 4025 raft_consensus.cc:1050] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 13d0f62249b34b01a3a47c6afe10e750 to VOTER
I20240503 02:13:48.710120 4096 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.710808 4025 consensus_queue.cc:237] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:48.712695 803 raft_consensus.cc:2932] T c46912d2d4434d4a8c3fa0c3c19ba14b P 05ef0a15decc468ea3a1af8828b0f630 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.720336 671 raft_consensus.cc:1261] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.721262 4096 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.723765 1067 raft_consensus.cc:1261] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.724313 4094 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.734792 31454 raft_consensus.cc:1261] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.735314 4094 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.738457 4096 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.744649 673 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.745450 31048 catalog_manager.cc:5484] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 7 to 8, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:48.749914 31323 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.761279 3046 raft_consensus.cc:1050] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 to VOTER
I20240503 02:13:48.762343 3046 consensus_queue.cc:237] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:48.765378 1972 raft_consensus.cc:1261] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.765090 4100 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.766556 4030 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.768566 803 raft_consensus.cc:1261] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.772378 4029 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.772689 936 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.773931 31452 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.774297 31048 catalog_manager.cc:5484] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:48.783551 31584 raft_consensus.cc:1261] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.784898 4032 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.791965 31191 consensus_queue.cc:237] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:48.795663 936 raft_consensus.cc:1261] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.796450 4094 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.799022 1067 raft_consensus.cc:1261] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.800014 4094 consensus_queue.cc:1035] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.801441 1066 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.810122 936 raft_consensus.cc:1261] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.810952 4031 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.810895 3676 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.812697 4121 tablet_replica.cc:330] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5: stopping tablet replica
I20240503 02:13:48.812925 4121 raft_consensus.cc:2229] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:48.813210 4121 pending_rounds.cc:62] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5: Trying to abort 1 pending ops.
I20240503 02:13:48.813347 4121 pending_rounds.cc:68] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5: Aborting op as it isn't in flight: id { term: 2 index: 9 } timestamp: 7023421146807025664 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c46912d2d4434d4a8c3fa0c3c19ba14b" old_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } } new_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } } }
I20240503 02:13:48.813834 4121 raft_consensus.cc:2894] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Aborting config change with OpId 2.9: Aborted: Op aborted
I20240503 02:13:48.813995 4121 raft_consensus.cc:2866] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:13:48.814121 4121 raft_consensus.cc:2258] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:48.815682 31038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4995796bdda64a7cbffa952a05a36673 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:48.817673 4121 ts_tablet_manager.cc:1875] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:48.818501 31454 consensus_queue.cc:237] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:48.820385 804 raft_consensus.cc:1261] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.820587 672 raft_consensus.cc:1261] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.820904 4113 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.820683 803 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.821262 4005 consensus_queue.cc:1035] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.824091 4121 ts_tablet_manager.cc:1888] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 02:13:48.824251 4121 log.cc:1196] T c46912d2d4434d4a8c3fa0c3c19ba14b P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/c46912d2d4434d4a8c3fa0c3c19ba14b
I20240503 02:13:48.824265 31044 catalog_manager.cc:5484] T 4995796bdda64a7cbffa952a05a36673 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index 8 to 9, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:48.825467 4030 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.826390 31038 catalog_manager.cc:4830] TS b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): tablet c46912d2d4434d4a8c3fa0c3c19ba14b (table rebalance_test_table_7 [id=7a99e03b1a8d47d1a25bbbd747784951]) successfully deleted
I20240503 02:13:48.826874 1066 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.828555 1972 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.828631 935 raft_consensus.cc:2932] T 4995796bdda64a7cbffa952a05a36673 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:48.832549 31583 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.833441 31452 raft_consensus.cc:1261] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.833968 4034 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.835072 673 raft_consensus.cc:1261] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.836355 4005 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.837771 801 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.838268 31044 catalog_manager.cc:5484] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.840095 3046 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.840250 31036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c3d978cd00449b8b3edd4c43b657960 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:48.840905 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet 4995796bdda64a7cbffa952a05a36673 with delete_type TABLET_DATA_TOMBSTONED (TS fbb7d16e67d94775b2881712419c0577 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41348
I20240503 02:13:48.841917 31046 catalog_manager.cc:5484] T 6c3d978cd00449b8b3edd4c43b657960 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 8 to 9, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:48.851137 4031 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:48.854578 801 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.855525 935 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:48.860918 31454 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:48.862612 672 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:48.865103 31049 catalog_manager.cc:5484] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 reported cstate change: config changed from index 7 to 8, 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:48.867339 31303 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c3d978cd00449b8b3edd4c43b657960 with delete_type TABLET_DATA_TOMBSTONED (TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55596
I20240503 02:13:48.872854 31324 consensus_queue.cc:237] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:48.872963 31323 consensus_queue.cc:237] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:48.875283 1972 raft_consensus.cc:1261] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.876061 3046 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.879479 4142 tablet_replica.cc:330] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: stopping tablet replica
I20240503 02:13:48.879745 4142 raft_consensus.cc:2229] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:48.879727 801 raft_consensus.cc:1261] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.879472 673 raft_consensus.cc:2932] T 6c3d978cd00449b8b3edd4c43b657960 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.879998 4142 raft_consensus.cc:2258] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:48.880156 3046 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.881168 4029 consensus_queue.cc:1035] T 7626c06dc67448f0b65f67b401d62c88 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.881810 4142 ts_tablet_manager.cc:1875] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:48.882433 935 raft_consensus.cc:1261] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.882732 801 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.882992 4029 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.884119 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7626c06dc67448f0b65f67b401d62c88 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:48.885294 4030 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:48.885829 935 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:48.888108 1972 raft_consensus.cc:2932] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.893378 31048 catalog_manager.cc:5484] T 3f8353133ccd4601a3761652546a33b5 P 4a3e3937b61447598fd76bc57f5d0939 reported cstate change: config changed from index 8 to 9, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:48.894454 4139 tablet_replica.cc:330] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577: stopping tablet replica
I20240503 02:13:48.894752 4139 raft_consensus.cc:2229] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:48.895074 4139 raft_consensus.cc:2258] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:48.896507 4139 ts_tablet_manager.cc:1875] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:48.896621 31050 catalog_manager.cc:5484] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 reported cstate change: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.898103 31171 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f8353133ccd4601a3761652546a33b5 with delete_type TABLET_DATA_TOMBSTONED (TS b52c8be37a2241ff9dc555aa0cebd5e5 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47568
I20240503 02:13:48.898485 4121 tablet_replica.cc:330] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5: stopping tablet replica
I20240503 02:13:48.898684 4121 raft_consensus.cc:2229] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:48.898960 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3f8353133ccd4601a3761652546a33b5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:48.899170 4121 raft_consensus.cc:2258] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:48.900159 4121 ts_tablet_manager.cc:1875] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:48.902199 31563 tablet_service.cc:1514] Processing DeleteTablet for tablet 7626c06dc67448f0b65f67b401d62c88 with delete_type TABLET_DATA_TOMBSTONED (TS b92ff49cec804b1f801c11b64e3259b3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36462
I20240503 02:13:48.902868 4139 ts_tablet_manager.cc:1888] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:48.903093 4139 log.cc:1196] T 4995796bdda64a7cbffa952a05a36673 P fbb7d16e67d94775b2881712419c0577: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/4995796bdda64a7cbffa952a05a36673
I20240503 02:13:48.903715 31323 consensus_queue.cc:237] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } }
I20240503 02:13:48.903980 4049 tablet_replica.cc:330] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3: stopping tablet replica
I20240503 02:13:48.904192 4049 raft_consensus.cc:2229] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:48.904438 4049 raft_consensus.cc:2258] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:48.904791 31036 catalog_manager.cc:4830] TS fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101): tablet 4995796bdda64a7cbffa952a05a36673 (table rebalance_test_table_9 [id=3a5935f3f7924b5083fcfc4ee7015893]) successfully deleted
I20240503 02:13:48.905812 4142 ts_tablet_manager.cc:1888] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:48.906473 935 raft_consensus.cc:1261] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.907287 673 raft_consensus.cc:1261] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.907450 4121 ts_tablet_manager.cc:1888] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:48.907581 4121 log.cc:1196] T 3f8353133ccd4601a3761652546a33b5 P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/3f8353133ccd4601a3761652546a33b5
I20240503 02:13:48.910143 1066 raft_consensus.cc:1261] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:48.907915 4030 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.910714 4032 consensus_queue.cc:1035] T 3f8353133ccd4601a3761652546a33b5 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.912573 4034 consensus_queue.cc:1035] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:48.913983 1066 raft_consensus.cc:2932] T 3f8353133ccd4601a3761652546a33b5 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:48.916807 4049 ts_tablet_manager.cc:1875] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:48.917222 4034 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:48.921383 673 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:48.924919 935 raft_consensus.cc:2932] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:48.930419 31038 catalog_manager.cc:4830] TS b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): tablet 3f8353133ccd4601a3761652546a33b5 (table rebalance_test_table_8 [id=dc947496e186485092d5d621503a4e6f]) successfully deleted
I20240503 02:13:48.932951 4142 log.cc:1196] T 6c3d978cd00449b8b3edd4c43b657960 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/6c3d978cd00449b8b3edd4c43b657960
I20240503 02:13:48.933017 4049 ts_tablet_manager.cc:1888] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:48.933167 4049 log.cc:1196] T 7626c06dc67448f0b65f67b401d62c88 P b92ff49cec804b1f801c11b64e3259b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/7626c06dc67448f0b65f67b401d62c88
I20240503 02:13:48.933917 31039 catalog_manager.cc:4830] TS b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849): tablet 7626c06dc67448f0b65f67b401d62c88 (table rebalance_test_table_5 [id=f64903d295f2485983d7a34ee454f2b9]) successfully deleted
I20240503 02:13:48.934509 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8c681bdcf3074c54b1d533cf1f7d2b75 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:48.934844 31037 catalog_manager.cc:4830] TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): tablet 6c3d978cd00449b8b3edd4c43b657960 (table rebalance_test_table_3 [id=e0a6e74497494a20be21d627357e4eb0]) successfully deleted
I20240503 02:13:48.931565 31044 catalog_manager.cc:5484] T 8c681bdcf3074c54b1d533cf1f7d2b75 P 13d0f62249b34b01a3a47c6afe10e750 reported cstate change: config changed from index 8 to 9, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } }
I20240503 02:13:48.935462 4029 raft_consensus.cc:1050] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 to VOTER
I20240503 02:13:48.936286 4029 consensus_queue.cc:237] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:48.939847 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c681bdcf3074c54b1d533cf1f7d2b75 with delete_type TABLET_DATA_TOMBSTONED (TS fbb7d16e67d94775b2881712419c0577 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41348
I20240503 02:13:48.940346 4139 tablet_replica.cc:330] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577: stopping tablet replica
I20240503 02:13:48.940589 4139 raft_consensus.cc:2229] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:48.940810 4139 raft_consensus.cc:2258] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:48.942828 4139 ts_tablet_manager.cc:1875] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:48.951828 4139 ts_tablet_manager.cc:1888] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:48.952100 4139 log.cc:1196] T 8c681bdcf3074c54b1d533cf1f7d2b75 P fbb7d16e67d94775b2881712419c0577: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/8c681bdcf3074c54b1d533cf1f7d2b75
I20240503 02:13:48.952925 31036 catalog_manager.cc:4830] TS fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101): tablet 8c681bdcf3074c54b1d533cf1f7d2b75 (table rebalance_test_table_4 [id=38897cba33374498b288d375d22641bd]) successfully deleted
I20240503 02:13:48.956219 1972 raft_consensus.cc:1261] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.956732 4030 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.957930 673 raft_consensus.cc:1261] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.958479 4030 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.968305 4025 raft_consensus.cc:1050] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 05ef0a15decc468ea3a1af8828b0f630 to VOTER
I20240503 02:13:48.969143 4025 consensus_queue.cc:237] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:48.969112 3046 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.971540 801 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.971623 31583 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.972103 4031 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.972404 4034 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.973101 935 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.973206 671 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.976426 4030 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.977361 4032 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.978667 801 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:48.982555 31050 catalog_manager.cc:5484] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 reported cstate change: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.984792 31191 raft_consensus.cc:1261] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:48.985376 4034 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:48.992415 1972 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:48.994133 31050 catalog_manager.cc:5484] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 reported cstate change: config changed from index 7 to 8, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:49.006654 31323 consensus_queue.cc:237] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:49.009958 31583 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:49.013705 31191 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } attrs { replace: true } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:49.015908 671 raft_consensus.cc:1261] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:49.016603 1972 raft_consensus.cc:1261] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:49.018313 3046 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:49.019196 3046 consensus_queue.cc:1035] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:49.023180 4034 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:49.027385 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b909a66859144f7b4b649d505c9645e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:49.029143 1972 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:49.029942 31050 catalog_manager.cc:5484] T 9b909a66859144f7b4b649d505c9645e P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 8 to 9, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:49.034454 31171 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b909a66859144f7b4b649d505c9645e with delete_type TABLET_DATA_TOMBSTONED (TS b52c8be37a2241ff9dc555aa0cebd5e5 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47568
I20240503 02:13:49.033864 671 raft_consensus.cc:2932] T 9b909a66859144f7b4b649d505c9645e P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:49.037943 4121 tablet_replica.cc:330] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5: stopping tablet replica
I20240503 02:13:49.038122 4121 raft_consensus.cc:2229] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:49.038327 4121 raft_consensus.cc:2258] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:49.039254 4121 ts_tablet_manager.cc:1875] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:49.045706 4121 ts_tablet_manager.cc:1888] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:49.045893 4121 log.cc:1196] T 9b909a66859144f7b4b649d505c9645e P b52c8be37a2241ff9dc555aa0cebd5e5: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/9b909a66859144f7b4b649d505c9645e
I20240503 02:13:49.046567 31038 catalog_manager.cc:4830] TS b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): tablet 9b909a66859144f7b4b649d505c9645e (table rebalance_test_table_6 [id=ad6bdc3b37e64f27824a7eacb47ca790]) successfully deleted
I20240503 02:13:49.046756 935 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:49.099339 31191 consensus_queue.cc:237] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:49.102149 1066 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:13:49.102699 4094 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:49.103143 31452 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:13:49.104457 4094 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:49.106156 4094 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.107434 671 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 2 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:49.108078 3950 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:49.109073 1066 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
W20240503 02:13:49.113855 31126 consensus_peers.cc:485] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 -> Peer 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601): Couldn't send request to peer 6ad20578b4a74bfabb6d1404de034a65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00e1dad5efda4e9eae55a8f9d80f0f96. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:49.113719 31452 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.119684 31048 catalog_manager.cc:5484] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 reported cstate change: config changed from index 7 to 8, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New cstate: current_term: 2 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.135248 31452 consensus_queue.cc:237] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:49.137406 801 raft_consensus.cc:1261] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:49.137483 31583 raft_consensus.cc:1261] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:49.137552 31387 consensus_peers.cc:485] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 -> Peer 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601): Couldn't send request to peer 6ad20578b4a74bfabb6d1404de034a65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c425b03a1c64d7ba7a14901d63d7037. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:49.137987 4113 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:49.138264 4096 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:49.140862 4113 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.141728 31583 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.142066 31452 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:49.142585 3950 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:49.144464 801 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.149746 31044 catalog_manager.cc:5484] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.165249 935 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:49.166066 3950 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:49.253222 31323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ce56ca4030a445db890303244caf308"
dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56958
I20240503 02:13:49.253422 31323 raft_consensus.cc:595] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:49.270313 31323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c91a7b003ed9411aba65a46d00c7d95b"
dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56986
I20240503 02:13:49.270548 31323 raft_consensus.cc:595] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:49.293567 3046 raft_consensus.cc:979] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: : Instructing follower b52c8be37a2241ff9dc555aa0cebd5e5 to start an election
I20240503 02:13:49.293704 3046 raft_consensus.cc:1067] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Signalling peer b52c8be37a2241ff9dc555aa0cebd5e5 to start an election
I20240503 02:13:49.294435 31191 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c91a7b003ed9411aba65a46d00c7d95b"
dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
from {username='slave'} at 127.23.104.194:59243
I20240503 02:13:49.294623 31191 raft_consensus.cc:482] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:49.294709 31191 raft_consensus.cc:3037] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:49.297267 31191 raft_consensus.cc:504] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:49.298588 1068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91a7b003ed9411aba65a46d00c7d95b" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "0429a57df6ba4f92a73e779b04b05040"
I20240503 02:13:49.298779 1068 raft_consensus.cc:3037] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:49.300033 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91a7b003ed9411aba65a46d00c7d95b" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "13d0f62249b34b01a3a47c6afe10e750"
I20240503 02:13:49.300221 671 raft_consensus.cc:3037] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:49.300712 1068 raft_consensus.cc:2454] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 2.
I20240503 02:13:49.302194 671 raft_consensus.cc:2454] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 2.
I20240503 02:13:49.302510 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91a7b003ed9411aba65a46d00c7d95b" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
I20240503 02:13:49.302560 31127 leader_election.cc:302] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0429a57df6ba4f92a73e779b04b05040, 13d0f62249b34b01a3a47c6afe10e750, b52c8be37a2241ff9dc555aa0cebd5e5; no voters:
I20240503 02:13:49.302716 31323 raft_consensus.cc:3032] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:49.302809 31323 raft_consensus.cc:729] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:49.302842 4094 raft_consensus.cc:2781] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:49.302987 4094 raft_consensus.cc:686] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 LEADER]: Becoming Leader. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Running, Role: LEADER
I20240503 02:13:49.303107 31323 consensus_queue.cc:260] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:49.303303 4094 consensus_queue.cc:237] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:49.303534 31323 raft_consensus.cc:3037] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:49.305579 31191 leader_election.cc:288] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 2 election: Requested vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543), 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401)
I20240503 02:13:49.306167 31323 raft_consensus.cc:2454] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 2.
I20240503 02:13:49.306362 31049 catalog_manager.cc:5484] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: term changed from 1 to 2, leader changed from cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) to b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193). New cstate: current_term: 2 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:49.316323 31323 consensus_queue.cc:237] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:49.318560 1068 raft_consensus.cc:1261] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:49.318584 31583 raft_consensus.cc:1261] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:49.318692 31257 consensus_peers.cc:485] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601): Couldn't send request to peer 6ad20578b4a74bfabb6d1404de034a65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb7e8fe39c024108be24bc4c144b247b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:49.319062 3046 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:49.319351 4030 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:49.322037 4030 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.322815 1068 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.328354 31049 catalog_manager.cc:5484] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:49.330767 31583 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.342321 31323 consensus_queue.cc:237] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:49.344367 31454 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240503 02:13:49.344518 31257 consensus_peers.cc:485] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 -> Peer 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601): Couldn't send request to peer 6ad20578b4a74bfabb6d1404de034a65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17be048393974fdf9811c5b8f8a97579. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:49.344523 935 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:13:49.345054 4030 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:49.345324 4031 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:49.347865 4030 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.348532 935 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.351862 31454 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.352999 31049 catalog_manager.cc:5484] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 reported cstate change: config changed from index 7 to 8, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New cstate: current_term: 2 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.365279 31454 consensus_queue.cc:237] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:49.367468 671 raft_consensus.cc:1261] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:49.367512 31583 raft_consensus.cc:1261] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:49.367568 31387 consensus_peers.cc:485] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 -> Peer 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601): Couldn't send request to peer 6ad20578b4a74bfabb6d1404de034a65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9df1587246a4df19fda13f4be2b1177. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:49.368017 4117 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:49.368306 4006 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:49.370962 4117 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.371760 671 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.376214 31048 catalog_manager.cc:5484] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:49.387676 31583 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } } }
I20240503 02:13:49.436942 31323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ce56ca4030a445db890303244caf308"
dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57034
I20240503 02:13:49.437152 31323 raft_consensus.cc:595] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:49.454203 3046 raft_consensus.cc:979] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: : Instructing follower 05ef0a15decc468ea3a1af8828b0f630 to start an election
I20240503 02:13:49.454393 3046 raft_consensus.cc:1067] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Signalling peer 05ef0a15decc468ea3a1af8828b0f630 to start an election
I20240503 02:13:49.455103 801 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ce56ca4030a445db890303244caf308"
dest_uuid: "05ef0a15decc468ea3a1af8828b0f630"
from {username='slave'} at 127.23.104.194:50489
I20240503 02:13:49.455353 801 raft_consensus.cc:482] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:49.455514 801 raft_consensus.cc:3037] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:49.457612 801 raft_consensus.cc:504] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:49.459596 801 leader_election.cc:288] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [CANDIDATE]: Term 2 election: Requested vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997)
I20240503 02:13:49.459892 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce56ca4030a445db890303244caf308" candidate_uuid: "05ef0a15decc468ea3a1af8828b0f630" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:49.460136 31583 raft_consensus.cc:3037] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:49.460664 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce56ca4030a445db890303244caf308" candidate_uuid: "05ef0a15decc468ea3a1af8828b0f630" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3"
I20240503 02:13:49.460877 31323 raft_consensus.cc:3032] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:49.460958 31323 raft_consensus.cc:729] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3, State: Running, Role: LEADER
I20240503 02:13:49.461210 31323 consensus_queue.cc:260] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:49.461632 31323 raft_consensus.cc:3037] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:49.462792 31583 raft_consensus.cc:2454] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ef0a15decc468ea3a1af8828b0f630 in term 2.
I20240503 02:13:49.463016 31584 raft_consensus.cc:1226] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Rejecting Update request from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 for earlier term 1. Current term is 2. Ops: []
I20240503 02:13:49.464634 31323 raft_consensus.cc:2454] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ef0a15decc468ea3a1af8828b0f630 in term 2.
I20240503 02:13:49.465356 738 leader_election.cc:302] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 05ef0a15decc468ea3a1af8828b0f630, b92ff49cec804b1f801c11b64e3259b3, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3; no voters:
I20240503 02:13:49.465737 3908 raft_consensus.cc:2781] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:49.469092 3908 raft_consensus.cc:686] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [term 2 LEADER]: Becoming Leader. State: Replica: 05ef0a15decc468ea3a1af8828b0f630, State: Running, Role: LEADER
I20240503 02:13:49.469842 3908 consensus_queue.cc:237] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:49.472102 935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce56ca4030a445db890303244caf308" candidate_uuid: "05ef0a15decc468ea3a1af8828b0f630" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4a3e3937b61447598fd76bc57f5d0939"
I20240503 02:13:49.472301 935 raft_consensus.cc:3037] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:49.473439 31048 catalog_manager.cc:5484] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 reported cstate change: term changed from 1 to 2, leader changed from cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) to 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198). New cstate: current_term: 2 leader_uuid: "05ef0a15decc468ea3a1af8828b0f630" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:49.474802 935 raft_consensus.cc:2454] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ef0a15decc468ea3a1af8828b0f630 in term 2.
I20240503 02:13:49.515161 4341 ts_tablet_manager.cc:902] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65: Initiating tablet copy from peer fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:49.516448 4341 tablet_copy_client.cc:308] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:39101
I20240503 02:13:49.517089 31474 tablet_copy_service.cc:140] P fbb7d16e67d94775b2881712419c0577: Received BeginTabletCopySession request for tablet 1c425b03a1c64d7ba7a14901d63d7037 from peer 6ad20578b4a74bfabb6d1404de034a65 ({username='slave'} at 127.23.104.201:38237)
I20240503 02:13:49.517246 31474 tablet_copy_service.cc:161] P fbb7d16e67d94775b2881712419c0577: Beginning new tablet copy session on tablet 1c425b03a1c64d7ba7a14901d63d7037 from peer 6ad20578b4a74bfabb6d1404de034a65 at {username='slave'} at 127.23.104.201:38237: session id = 6ad20578b4a74bfabb6d1404de034a65-1c425b03a1c64d7ba7a14901d63d7037
I20240503 02:13:49.519261 31474 tablet_copy_source_session.cc:205] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:49.520395 4341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c425b03a1c64d7ba7a14901d63d7037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:49.524746 4341 tablet_copy_client.cc:783] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:49.524983 4341 tablet_copy_client.cc:647] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:49.526291 4345 tablet_copy_client.cc:1002] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:49.670452 4346 ts_tablet_manager.cc:902] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65: Initiating tablet copy from peer b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019)
I20240503 02:13:49.671630 4346 tablet_copy_client.cc:308] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.193:44019
I20240503 02:13:49.678776 31213 tablet_copy_service.cc:140] P b52c8be37a2241ff9dc555aa0cebd5e5: Received BeginTabletCopySession request for tablet 00e1dad5efda4e9eae55a8f9d80f0f96 from peer 6ad20578b4a74bfabb6d1404de034a65 ({username='slave'} at 127.23.104.201:49095)
I20240503 02:13:49.678980 31213 tablet_copy_service.cc:161] P b52c8be37a2241ff9dc555aa0cebd5e5: Beginning new tablet copy session on tablet 00e1dad5efda4e9eae55a8f9d80f0f96 from peer 6ad20578b4a74bfabb6d1404de034a65 at {username='slave'} at 127.23.104.201:49095: session id = 6ad20578b4a74bfabb6d1404de034a65-00e1dad5efda4e9eae55a8f9d80f0f96
I20240503 02:13:49.681912 31213 tablet_copy_source_session.cc:205] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:49.683120 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e1dad5efda4e9eae55a8f9d80f0f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:49.687682 4346 tablet_copy_client.cc:783] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:49.687928 4346 tablet_copy_client.cc:647] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:49.689498 4350 tablet_copy_client.cc:1002] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:49.704115 31323 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:49.704784 4094 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:49.737674 31584 consensus_queue.cc:237] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:49.738436 1068 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:49.739058 4094 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:49.739428 935 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:49.739668 671 raft_consensus.cc:1261] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 2 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:49.740284 3957 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:49.740751 3953 consensus_queue.cc:1035] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:49.747756 4383 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:49.750209 935 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:49.753793 31039 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9000ce76eb154434a88b52e357497767 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:49.754712 31049 catalog_manager.cc:5484] T 9000ce76eb154434a88b52e357497767 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 8 to 10, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New cstate: current_term: 2 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:49.755215 671 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:49.755125 673 raft_consensus.cc:2932] T 9000ce76eb154434a88b52e357497767 P 13d0f62249b34b01a3a47c6afe10e750 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:49.755883 4100 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:49.759631 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet 9000ce76eb154434a88b52e357497767 with delete_type TABLET_DATA_TOMBSTONED (TS fbb7d16e67d94775b2881712419c0577 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41348
I20240503 02:13:49.772794 4390 tablet_replica.cc:330] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577: stopping tablet replica
I20240503 02:13:49.773049 4390 raft_consensus.cc:2229] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:49.773269 4390 raft_consensus.cc:2258] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:49.774863 4388 ts_tablet_manager.cc:902] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65: Initiating tablet copy from peer fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:13:49.778374 4390 ts_tablet_manager.cc:1875] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:49.788179 4391 ts_tablet_manager.cc:902] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:49.799429 4391 tablet_copy_client.cc:308] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:49.800442 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet 17be048393974fdf9811c5b8f8a97579 from peer 6ad20578b4a74bfabb6d1404de034a65 ({username='slave'} at 127.23.104.201:50129)
I20240503 02:13:49.800635 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet 17be048393974fdf9811c5b8f8a97579 from peer 6ad20578b4a74bfabb6d1404de034a65 at {username='slave'} at 127.23.104.201:50129: session id = 6ad20578b4a74bfabb6d1404de034a65-17be048393974fdf9811c5b8f8a97579
I20240503 02:13:49.801318 4390 ts_tablet_manager.cc:1888] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 02:13:49.801414 4390 log.cc:1196] T 9000ce76eb154434a88b52e357497767 P fbb7d16e67d94775b2881712419c0577: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/9000ce76eb154434a88b52e357497767
I20240503 02:13:49.802116 31036 catalog_manager.cc:4830] TS fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101): tablet 9000ce76eb154434a88b52e357497767 (table rebalance_test_table_5 [id=f64903d295f2485983d7a34ee454f2b9]) successfully deleted
I20240503 02:13:49.807289 31344 tablet_copy_source_session.cc:205] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:49.808195 4388 tablet_copy_client.cc:308] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:39101
I20240503 02:13:49.808857 31474 tablet_copy_service.cc:140] P fbb7d16e67d94775b2881712419c0577: Received BeginTabletCopySession request for tablet d9df1587246a4df19fda13f4be2b1177 from peer 6ad20578b4a74bfabb6d1404de034a65 ({username='slave'} at 127.23.104.201:38237)
I20240503 02:13:49.808949 4391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17be048393974fdf9811c5b8f8a97579. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:49.809032 31474 tablet_copy_service.cc:161] P fbb7d16e67d94775b2881712419c0577: Beginning new tablet copy session on tablet d9df1587246a4df19fda13f4be2b1177 from peer 6ad20578b4a74bfabb6d1404de034a65 at {username='slave'} at 127.23.104.201:38237: session id = 6ad20578b4a74bfabb6d1404de034a65-d9df1587246a4df19fda13f4be2b1177
I20240503 02:13:49.810900 31474 tablet_copy_source_session.cc:205] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:49.811940 4388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9df1587246a4df19fda13f4be2b1177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:49.814034 4391 tablet_copy_client.cc:783] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:49.814371 4391 tablet_copy_client.cc:647] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:49.816865 4403 tablet_copy_client.cc:1002] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:49.817867 4388 tablet_copy_client.cc:783] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:49.818099 4388 tablet_copy_client.cc:647] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:49.819511 4401 tablet_copy_client.cc:1002] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:49.857594 31323 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Refusing update from remote peer 05ef0a15decc468ea3a1af8828b0f630: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:49.858274 3908 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:49.884361 4414 ts_tablet_manager.cc:902] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65: Initiating tablet copy from peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179)
I20240503 02:13:49.885668 4414 tablet_copy_client.cc:308] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:46179
I20240503 02:13:49.886390 31344 tablet_copy_service.cc:140] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Received BeginTabletCopySession request for tablet cb7e8fe39c024108be24bc4c144b247b from peer 6ad20578b4a74bfabb6d1404de034a65 ({username='slave'} at 127.23.104.201:50129)
I20240503 02:13:49.886566 31344 tablet_copy_service.cc:161] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Beginning new tablet copy session on tablet cb7e8fe39c024108be24bc4c144b247b from peer 6ad20578b4a74bfabb6d1404de034a65 at {username='slave'} at 127.23.104.201:50129: session id = 6ad20578b4a74bfabb6d1404de034a65-cb7e8fe39c024108be24bc4c144b247b
I20240503 02:13:49.886663 935 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Refusing update from remote peer 05ef0a15decc468ea3a1af8828b0f630: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:49.887305 3908 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:49.889250 31344 tablet_copy_source_session.cc:205] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:49.890986 4414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7e8fe39c024108be24bc4c144b247b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:49.899158 4414 tablet_copy_client.cc:783] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:49.899483 4414 tablet_copy_client.cc:647] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:49.901793 31584 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Refusing update from remote peer 05ef0a15decc468ea3a1af8828b0f630: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:49.902799 3908 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:49.910241 4418 tablet_copy_client.cc:1002] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:50.177268 31191 consensus_queue.cc:237] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:13:50.179054 1068 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:50.179399 671 raft_consensus.cc:1261] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:50.179949 4094 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:50.180275 4384 consensus_queue.cc:1035] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:50.182461 4100 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:50.183023 671 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:50.183040 1068 raft_consensus.cc:2932] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:50.186300 31038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c91a7b003ed9411aba65a46d00c7d95b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:50.187749 31048 catalog_manager.cc:5484] T c91a7b003ed9411aba65a46d00c7d95b P 13d0f62249b34b01a3a47c6afe10e750 reported cstate change: config changed from index 8 to 10, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New cstate: current_term: 2 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } }
I20240503 02:13:50.191944 31303 tablet_service.cc:1514] Processing DeleteTablet for tablet c91a7b003ed9411aba65a46d00c7d95b with delete_type TABLET_DATA_TOMBSTONED (TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55596
I20240503 02:13:50.192600 4503 tablet_replica.cc:330] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: stopping tablet replica
I20240503 02:13:50.192817 4503 raft_consensus.cc:2229] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:50.193020 4503 raft_consensus.cc:2258] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:50.193759 4503 ts_tablet_manager.cc:1875] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:50.198338 4503 ts_tablet_manager.cc:1888] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 02:13:50.198432 4503 log.cc:1196] T c91a7b003ed9411aba65a46d00c7d95b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/c91a7b003ed9411aba65a46d00c7d95b
I20240503 02:13:50.199129 31037 catalog_manager.cc:4830] TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): tablet c91a7b003ed9411aba65a46d00c7d95b (table rebalance_test_table_1 [id=d3a62fd4d46c4ba082abbec8f97c84e0]) successfully deleted
I20240503 02:13:50.401196 801 consensus_queue.cc:237] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } }
I20240503 02:13:50.403231 31584 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Refusing update from remote peer 05ef0a15decc468ea3a1af8828b0f630: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:50.403354 935 raft_consensus.cc:1261] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Refusing update from remote peer 05ef0a15decc468ea3a1af8828b0f630: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:50.403707 3908 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:50.404029 4421 consensus_queue.cc:1035] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:50.406437 4337 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:50.406767 935 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:50.412837 31584 raft_consensus.cc:2932] T 6ce56ca4030a445db890303244caf308 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } }
I20240503 02:13:50.415045 31038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6ce56ca4030a445db890303244caf308 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:50.418275 31044 catalog_manager.cc:5484] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 reported cstate change: config changed from index 8 to 10, VOTER cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) evicted. New cstate: current_term: 2 leader_uuid: "05ef0a15decc468ea3a1af8828b0f630" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:50.422994 31303 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ce56ca4030a445db890303244caf308 with delete_type TABLET_DATA_TOMBSTONED (TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55596
I20240503 02:13:50.424209 4503 tablet_replica.cc:330] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: stopping tablet replica
I20240503 02:13:50.424403 4503 raft_consensus.cc:2229] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:50.424645 4503 raft_consensus.cc:2258] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:50.426736 4503 ts_tablet_manager.cc:1875] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:50.434443 4503 ts_tablet_manager.cc:1888] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 02:13:50.434561 4503 log.cc:1196] T 6ce56ca4030a445db890303244caf308 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/6ce56ca4030a445db890303244caf308
I20240503 02:13:50.435323 31037 catalog_manager.cc:4830] TS cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): tablet 6ce56ca4030a445db890303244caf308 (table rebalance_test_table_6 [id=ad6bdc3b37e64f27824a7eacb47ca790]) successfully deleted
I20240503 02:13:51.026612 4341 tablet_copy_client.cc:523] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:51.029147 4341 tablet_bootstrap.cc:492] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap starting.
I20240503 02:13:51.061314 4341 tablet_bootstrap.cc:492] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:51.061669 4341 tablet_bootstrap.cc:492] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap complete.
I20240503 02:13:51.061997 4341 ts_tablet_manager.cc:1367] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65: Time spent bootstrapping tablet: real 0.033s user 0.032s sys 0.000s
I20240503 02:13:51.062712 4341 raft_consensus.cc:348] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:51.062942 4341 raft_consensus.cc:729] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ad20578b4a74bfabb6d1404de034a65, State: Initialized, Role: LEARNER
I20240503 02:13:51.063119 4341 consensus_queue.cc:260] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:51.063709 4341 ts_tablet_manager.cc:1398] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:51.064392 31474 tablet_copy_service.cc:342] P fbb7d16e67d94775b2881712419c0577: Request end of tablet copy session 6ad20578b4a74bfabb6d1404de034a65-1c425b03a1c64d7ba7a14901d63d7037 received from {username='slave'} at 127.23.104.201:38237
I20240503 02:13:51.064527 31474 tablet_copy_service.cc:433] P fbb7d16e67d94775b2881712419c0577: ending tablet copy session 6ad20578b4a74bfabb6d1404de034a65-1c425b03a1c64d7ba7a14901d63d7037 on tablet 1c425b03a1c64d7ba7a14901d63d7037 with peer 6ad20578b4a74bfabb6d1404de034a65
I20240503 02:13:51.189965 4346 tablet_copy_client.cc:523] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:51.192719 4346 tablet_bootstrap.cc:492] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap starting.
I20240503 02:13:51.224283 4346 tablet_bootstrap.cc:492] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:51.224632 4346 tablet_bootstrap.cc:492] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap complete.
I20240503 02:13:51.224853 4346 ts_tablet_manager.cc:1367] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65: Time spent bootstrapping tablet: real 0.032s user 0.031s sys 0.001s
I20240503 02:13:51.225477 4346 raft_consensus.cc:348] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:51.225668 4346 raft_consensus.cc:729] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6ad20578b4a74bfabb6d1404de034a65, State: Initialized, Role: LEARNER
I20240503 02:13:51.225855 4346 consensus_queue.cc:260] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:51.226402 4346 ts_tablet_manager.cc:1398] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:13:51.227196 31213 tablet_copy_service.cc:342] P b52c8be37a2241ff9dc555aa0cebd5e5: Request end of tablet copy session 6ad20578b4a74bfabb6d1404de034a65-00e1dad5efda4e9eae55a8f9d80f0f96 received from {username='slave'} at 127.23.104.201:49095
I20240503 02:13:51.227331 31213 tablet_copy_service.cc:433] P b52c8be37a2241ff9dc555aa0cebd5e5: ending tablet copy session 6ad20578b4a74bfabb6d1404de034a65-00e1dad5efda4e9eae55a8f9d80f0f96 on tablet 00e1dad5efda4e9eae55a8f9d80f0f96 with peer 6ad20578b4a74bfabb6d1404de034a65
I20240503 02:13:51.317227 4391 tablet_copy_client.cc:523] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:51.319795 4391 tablet_bootstrap.cc:492] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap starting.
I20240503 02:13:51.319885 4388 tablet_copy_client.cc:523] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:51.323105 4388 tablet_bootstrap.cc:492] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap starting.
I20240503 02:13:51.369565 4391 tablet_bootstrap.cc:492] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:51.369993 4391 tablet_bootstrap.cc:492] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap complete.
I20240503 02:13:51.370260 4391 ts_tablet_manager.cc:1367] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65: Time spent bootstrapping tablet: real 0.051s user 0.050s sys 0.001s
I20240503 02:13:51.371039 4391 raft_consensus.cc:348] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:51.371299 4391 raft_consensus.cc:729] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6ad20578b4a74bfabb6d1404de034a65, State: Initialized, Role: LEARNER
I20240503 02:13:51.371532 4391 consensus_queue.cc:260] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:51.372043 4388 tablet_bootstrap.cc:492] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:51.372150 4391 ts_tablet_manager.cc:1398] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:51.372493 4388 tablet_bootstrap.cc:492] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap complete.
I20240503 02:13:51.372746 4388 ts_tablet_manager.cc:1367] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65: Time spent bootstrapping tablet: real 0.050s user 0.048s sys 0.003s
I20240503 02:13:51.372928 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 6ad20578b4a74bfabb6d1404de034a65-17be048393974fdf9811c5b8f8a97579 received from {username='slave'} at 127.23.104.201:50129
I20240503 02:13:51.373051 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 6ad20578b4a74bfabb6d1404de034a65-17be048393974fdf9811c5b8f8a97579 on tablet 17be048393974fdf9811c5b8f8a97579 with peer 6ad20578b4a74bfabb6d1404de034a65
I20240503 02:13:51.373415 4388 raft_consensus.cc:348] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:51.373628 4388 raft_consensus.cc:729] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ad20578b4a74bfabb6d1404de034a65, State: Initialized, Role: LEARNER
I20240503 02:13:51.373809 4388 consensus_queue.cc:260] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:51.374337 4388 ts_tablet_manager.cc:1398] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:13:51.375131 31474 tablet_copy_service.cc:342] P fbb7d16e67d94775b2881712419c0577: Request end of tablet copy session 6ad20578b4a74bfabb6d1404de034a65-d9df1587246a4df19fda13f4be2b1177 received from {username='slave'} at 127.23.104.201:38237
I20240503 02:13:51.375267 31474 tablet_copy_service.cc:433] P fbb7d16e67d94775b2881712419c0577: ending tablet copy session 6ad20578b4a74bfabb6d1404de034a65-d9df1587246a4df19fda13f4be2b1177 on tablet d9df1587246a4df19fda13f4be2b1177 with peer 6ad20578b4a74bfabb6d1404de034a65
I20240503 02:13:51.410609 4414 tablet_copy_client.cc:523] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:51.412987 4414 tablet_bootstrap.cc:492] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap starting.
I20240503 02:13:51.444931 4414 tablet_bootstrap.cc:492] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:51.445333 4414 tablet_bootstrap.cc:492] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65: Bootstrap complete.
I20240503 02:13:51.445600 4414 ts_tablet_manager.cc:1367] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.004s
I20240503 02:13:51.446182 4414 raft_consensus.cc:348] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:51.446385 4414 raft_consensus.cc:729] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ad20578b4a74bfabb6d1404de034a65, State: Initialized, Role: LEARNER
I20240503 02:13:51.446519 4414 consensus_queue.cc:260] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: true } }
I20240503 02:13:51.446951 4414 ts_tablet_manager.cc:1398] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:13:51.447695 31344 tablet_copy_service.cc:342] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Request end of tablet copy session 6ad20578b4a74bfabb6d1404de034a65-cb7e8fe39c024108be24bc4c144b247b received from {username='slave'} at 127.23.104.201:50129
I20240503 02:13:51.447863 31344 tablet_copy_service.cc:433] P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: ending tablet copy session 6ad20578b4a74bfabb6d1404de034a65-cb7e8fe39c024108be24bc4c144b247b on tablet cb7e8fe39c024108be24bc4c144b247b with peer 6ad20578b4a74bfabb6d1404de034a65
I20240503 02:13:51.608366 1970 raft_consensus.cc:1203] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:51.620198 1970 raft_consensus.cc:1203] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:51.648104 1970 raft_consensus.cc:1203] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 02:13:51.671222 1970 raft_consensus.cc:1203] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:51.766849 1970 raft_consensus.cc:1203] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 02:13:52.123551 4439 raft_consensus.cc:1050] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577: attempting to promote NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 to VOTER
I20240503 02:13:52.124217 4439 consensus_queue.cc:237] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:52.126477 1970 raft_consensus.cc:1261] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:52.126796 801 raft_consensus.cc:1261] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:52.127038 4734 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:52.127274 31584 raft_consensus.cc:1261] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:52.127422 4439 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:52.127889 4732 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:52.131688 4732 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.132314 31584 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.133649 1970 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.137785 801 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.137989 31044 catalog_manager.cc:5484] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.144843 31454 consensus_queue.cc:237] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:52.146224 1970 raft_consensus.cc:1261] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:52.146243 801 raft_consensus.cc:1261] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:52.146737 4439 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:52.147037 4734 consensus_queue.cc:1035] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:52.149207 4734 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.149817 1970 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.149942 801 raft_consensus.cc:2932] T 1c425b03a1c64d7ba7a14901d63d7037 P 05ef0a15decc468ea3a1af8828b0f630 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.152701 31036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c425b03a1c64d7ba7a14901d63d7037 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:52.154358 31050 catalog_manager.cc:5484] T 1c425b03a1c64d7ba7a14901d63d7037 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:52.157935 31563 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c425b03a1c64d7ba7a14901d63d7037 with delete_type TABLET_DATA_TOMBSTONED (TS b92ff49cec804b1f801c11b64e3259b3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36462
I20240503 02:13:52.172520 4743 tablet_replica.cc:330] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3: stopping tablet replica
I20240503 02:13:52.173658 4439 raft_consensus.cc:1050] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577: attempting to promote NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 to VOTER
I20240503 02:13:52.173631 4743 raft_consensus.cc:2229] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:52.174423 4743 raft_consensus.cc:2258] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:52.174758 4439 consensus_queue.cc:237] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:52.175829 4743 ts_tablet_manager.cc:1875] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:52.177403 1970 raft_consensus.cc:1261] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:52.177471 671 raft_consensus.cc:1261] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:52.178169 4705 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:52.178550 4732 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:52.182341 4651 raft_consensus.cc:1050] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5: attempting to promote NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 to VOTER
I20240503 02:13:52.183286 4651 consensus_queue.cc:237] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:52.187690 31584 raft_consensus.cc:1261] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:52.188450 4439 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:52.189765 1972 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 LEARNER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:52.190443 4744 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:52.197762 1068 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:52.198486 4742 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:52.201452 31454 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:52.202042 4742 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:52.203047 4439 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.203676 673 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.209596 31584 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.210912 4501 raft_consensus.cc:1050] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 to VOTER
I20240503 02:13:52.211803 4501 consensus_queue.cc:237] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:52.214969 4744 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.216161 31044 catalog_manager.cc:5484] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.217646 4743 ts_tablet_manager.cc:1888] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:52.217934 4743 log.cc:1196] T 1c425b03a1c64d7ba7a14901d63d7037 P b92ff49cec804b1f801c11b64e3259b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/1c425b03a1c64d7ba7a14901d63d7037
I20240503 02:13:52.220055 1972 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.220002 31454 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.223599 31039 catalog_manager.cc:4830] TS b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849): tablet 1c425b03a1c64d7ba7a14901d63d7037 (table rebalance_test_table_4 [id=38897cba33374498b288d375d22641bd]) successfully deleted
I20240503 02:13:52.227281 31048 catalog_manager.cc:5484] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 reported cstate change: config changed from index 8 to 9, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.245579 1066 raft_consensus.cc:1261] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:52.245605 1972 raft_consensus.cc:1261] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [term 1 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:52.246217 4704 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:52.246865 4704 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:52.249315 1068 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.250612 1970 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.253618 31454 consensus_queue.cc:237] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:52.255818 1970 raft_consensus.cc:1261] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:52.256340 671 raft_consensus.cc:1261] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer fbb7d16e67d94775b2881712419c0577: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:52.256911 4439 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:52.257195 4705 consensus_queue.cc:1035] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:52.257833 31193 consensus_queue.cc:237] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:52.260188 1970 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:52.259914 4732 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.260411 671 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.264165 1068 raft_consensus.cc:1261] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:52.264129 4651 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:52.264792 31036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d9df1587246a4df19fda13f4be2b1177 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:52.264851 4744 consensus_queue.cc:1035] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:52.266211 4761 raft_consensus.cc:1050] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: attempting to promote NON_VOTER 6ad20578b4a74bfabb6d1404de034a65 to VOTER
I20240503 02:13:52.266968 4761 consensus_queue.cc:237] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:52.267072 4744 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.268942 31048 catalog_manager.cc:5484] T d9df1587246a4df19fda13f4be2b1177 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "fbb7d16e67d94775b2881712419c0577" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:52.271667 1971 raft_consensus.cc:2932] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.272655 31038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00e1dad5efda4e9eae55a8f9d80f0f96 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:52.272926 31563 tablet_service.cc:1514] Processing DeleteTablet for tablet d9df1587246a4df19fda13f4be2b1177 with delete_type TABLET_DATA_TOMBSTONED (TS b92ff49cec804b1f801c11b64e3259b3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36462
I20240503 02:13:52.273926 4743 tablet_replica.cc:330] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3: stopping tablet replica
I20240503 02:13:52.274125 4743 raft_consensus.cc:2229] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:52.274361 4743 raft_consensus.cc:2258] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:52.274140 1068 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.275357 4743 ts_tablet_manager.cc:1875] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:52.281265 1971 raft_consensus.cc:2932] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.282049 31044 catalog_manager.cc:5484] T 00e1dad5efda4e9eae55a8f9d80f0f96 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index 9 to 10, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New cstate: current_term: 2 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:52.284128 4743 ts_tablet_manager.cc:1888] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:52.284322 4743 log.cc:1196] T d9df1587246a4df19fda13f4be2b1177 P b92ff49cec804b1f801c11b64e3259b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/d9df1587246a4df19fda13f4be2b1177
I20240503 02:13:52.284538 31584 raft_consensus.cc:1261] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:52.285231 4704 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:52.285342 31039 catalog_manager.cc:4830] TS b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849): tablet d9df1587246a4df19fda13f4be2b1177 (table rebalance_test_table_9 [id=3a5935f3f7924b5083fcfc4ee7015893]) successfully deleted
I20240503 02:13:52.288568 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet 00e1dad5efda4e9eae55a8f9d80f0f96 with delete_type TABLET_DATA_TOMBSTONED (TS fbb7d16e67d94775b2881712419c0577 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41348
I20240503 02:13:52.288286 4704 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.289343 31584 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.294463 31049 catalog_manager.cc:5484] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:52.294507 1971 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.305056 1970 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 LEARNER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:52.307279 935 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:52.307286 4772 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:52.307902 4704 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:52.307849 1066 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.311074 31323 consensus_queue.cc:237] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:52.313642 1068 raft_consensus.cc:1261] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:52.313999 1971 raft_consensus.cc:1261] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:52.314216 4501 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:52.314713 4501 consensus_queue.cc:1035] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:52.316367 4771 tablet_replica.cc:330] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577: stopping tablet replica
I20240503 02:13:52.316632 4771 raft_consensus.cc:2229] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:52.317416 4771 raft_consensus.cc:2258] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:52.317905 4704 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.318630 31454 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:52.319175 4773 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:52.321660 4771 ts_tablet_manager.cc:1875] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:52.324875 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cb7e8fe39c024108be24bc4c144b247b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:52.326668 1068 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.326642 4501 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.327950 31454 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.328080 31048 catalog_manager.cc:5484] T cb7e8fe39c024108be24bc4c144b247b P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:52.334659 31563 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7e8fe39c024108be24bc4c144b247b with delete_type TABLET_DATA_TOMBSTONED (TS b92ff49cec804b1f801c11b64e3259b3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36462
I20240503 02:13:52.334995 4743 tablet_replica.cc:330] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3: stopping tablet replica
I20240503 02:13:52.335202 4743 raft_consensus.cc:2229] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:52.334935 31049 catalog_manager.cc:5484] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 reported cstate change: config changed from index 8 to 9, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.335541 4743 raft_consensus.cc:2258] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:52.335542 1970 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.336483 4743 ts_tablet_manager.cc:1875] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:52.336992 935 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } attrs { replace: true } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.344447 1971 raft_consensus.cc:2932] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.347136 4743 ts_tablet_manager.cc:1888] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:52.347263 4743 log.cc:1196] T cb7e8fe39c024108be24bc4c144b247b P b92ff49cec804b1f801c11b64e3259b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/cb7e8fe39c024108be24bc4c144b247b
I20240503 02:13:52.348233 31039 catalog_manager.cc:4830] TS b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849): tablet cb7e8fe39c024108be24bc4c144b247b (table rebalance_test_table_7 [id=7a99e03b1a8d47d1a25bbbd747784951]) successfully deleted
I20240503 02:13:52.351408 4771 ts_tablet_manager.cc:1888] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 02:13:52.351611 4771 log.cc:1196] T 00e1dad5efda4e9eae55a8f9d80f0f96 P fbb7d16e67d94775b2881712419c0577: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/00e1dad5efda4e9eae55a8f9d80f0f96
I20240503 02:13:52.354307 31323 consensus_queue.cc:237] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:13:52.355955 31036 catalog_manager.cc:4830] TS fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101): tablet 00e1dad5efda4e9eae55a8f9d80f0f96 (table rebalance_test_table_0 [id=cc3c62c772ec4610892f67e8c6d52379]) successfully deleted
I20240503 02:13:52.357314 1971 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:52.357767 4773 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:52.359329 935 raft_consensus.cc:1261] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Refusing update from remote peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:52.359922 4773 consensus_queue.cc:1035] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:52.360397 4772 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.360960 1971 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.361392 936 raft_consensus.cc:2932] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } } }
I20240503 02:13:52.364419 31037 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 17be048393974fdf9811c5b8f8a97579 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:52.365770 31044 catalog_manager.cc:5484] T 17be048393974fdf9811c5b8f8a97579 P cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 reported cstate change: config changed from index 9 to 10, VOTER fbb7d16e67d94775b2881712419c0577 (127.23.104.195) evicted. New cstate: current_term: 2 leader_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:52.369719 31433 tablet_service.cc:1514] Processing DeleteTablet for tablet 17be048393974fdf9811c5b8f8a97579 with delete_type TABLET_DATA_TOMBSTONED (TS fbb7d16e67d94775b2881712419c0577 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41348
I20240503 02:13:52.370028 4771 tablet_replica.cc:330] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577: stopping tablet replica
I20240503 02:13:52.370189 4771 raft_consensus.cc:2229] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:52.370501 4771 raft_consensus.cc:2258] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:52.371424 4771 ts_tablet_manager.cc:1875] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:52.376031 4771 ts_tablet_manager.cc:1888] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 02:13:52.376139 4771 log.cc:1196] T 17be048393974fdf9811c5b8f8a97579 P fbb7d16e67d94775b2881712419c0577: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/17be048393974fdf9811c5b8f8a97579
I20240503 02:13:52.376770 31036 catalog_manager.cc:4830] TS fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101): tablet 17be048393974fdf9811c5b8f8a97579 (table rebalance_test_table_3 [id=e0a6e74497494a20be21d627357e4eb0]) successfully deleted
I20240503 02:13:52.528388 1048 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:52.568610 784 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:52.585709 652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:52.589123 1951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:52.610447 31171 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:52.655965 31563 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:52.668622 31303 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:52.679162 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:52.699568 31433 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:52.786404 31584 consensus_queue.cc:237] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:52.789227 31193 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:52.789294 31518 consensus_peers.cc:485] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 -> Peer 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997): Couldn't send request to peer 4a3e3937b61447598fd76bc57f5d0939. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a5406bce7014bc2a7a1d727cb4aa191. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:52.789732 4849 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:52.790098 673 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:52.790755 4849 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:52.792899 4849 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:52.793619 31193 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:52.794056 673 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } } }
I20240503 02:13:52.797865 31047 catalog_manager.cc:5484] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 6 to 7, NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) added. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:53.226454 4870 ts_tablet_manager.cc:902] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939: Initiating tablet copy from peer b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849)
I20240503 02:13:53.227703 4870 tablet_copy_client.cc:308] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:39849
I20240503 02:13:53.228397 31604 tablet_copy_service.cc:140] P b92ff49cec804b1f801c11b64e3259b3: Received BeginTabletCopySession request for tablet 8a5406bce7014bc2a7a1d727cb4aa191 from peer 4a3e3937b61447598fd76bc57f5d0939 ({username='slave'} at 127.23.104.199:40937)
I20240503 02:13:53.228562 31604 tablet_copy_service.cc:161] P b92ff49cec804b1f801c11b64e3259b3: Beginning new tablet copy session on tablet 8a5406bce7014bc2a7a1d727cb4aa191 from peer 4a3e3937b61447598fd76bc57f5d0939 at {username='slave'} at 127.23.104.199:40937: session id = 4a3e3937b61447598fd76bc57f5d0939-8a5406bce7014bc2a7a1d727cb4aa191
I20240503 02:13:53.231309 31604 tablet_copy_source_session.cc:205] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:53.232532 4870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a5406bce7014bc2a7a1d727cb4aa191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:53.236724 4870 tablet_copy_client.cc:783] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:53.236985 4870 tablet_copy_client.cc:647] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:53.238479 4874 tablet_copy_client.cc:1002] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Injecting latency into file download: 1500
I20240503 02:13:54.738932 4870 tablet_copy_client.cc:523] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:54.741742 4870 tablet_bootstrap.cc:492] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap starting.
I20240503 02:13:54.772914 4870 tablet_bootstrap.cc:492] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:54.773298 4870 tablet_bootstrap.cc:492] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939: Bootstrap complete.
I20240503 02:13:54.773587 4870 ts_tablet_manager.cc:1367] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.005s
I20240503 02:13:54.774312 4870 raft_consensus.cc:348] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:54.774538 4870 raft_consensus.cc:729] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a3e3937b61447598fd76bc57f5d0939, State: Initialized, Role: LEARNER
I20240503 02:13:54.774750 4870 consensus_queue.cc:260] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: NON_VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: true } }
I20240503 02:13:54.775513 4870 ts_tablet_manager.cc:1398] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:54.776307 31604 tablet_copy_service.cc:342] P b92ff49cec804b1f801c11b64e3259b3: Request end of tablet copy session 4a3e3937b61447598fd76bc57f5d0939-8a5406bce7014bc2a7a1d727cb4aa191 received from {username='slave'} at 127.23.104.199:40937
I20240503 02:13:54.776429 31604 tablet_copy_service.cc:433] P b92ff49cec804b1f801c11b64e3259b3: ending tablet copy session 4a3e3937b61447598fd76bc57f5d0939-8a5406bce7014bc2a7a1d727cb4aa191 on tablet 8a5406bce7014bc2a7a1d727cb4aa191 with peer 4a3e3937b61447598fd76bc57f5d0939
I20240503 02:13:55.162696 936 raft_consensus.cc:1203] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:55.566143 4938 raft_consensus.cc:1050] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3: attempting to promote NON_VOTER 4a3e3937b61447598fd76bc57f5d0939 to VOTER
I20240503 02:13:55.566970 4938 consensus_queue.cc:237] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:55.569443 936 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 LEARNER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:55.569473 31191 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:55.569614 673 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Refusing update from remote peer b92ff49cec804b1f801c11b64e3259b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:55.570055 4939 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:55.570304 4938 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:55.570533 4886 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:55.573359 4938 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:55.573956 31191 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:55.573992 936 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:55.578774 31049 catalog_manager.cc:5484] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 reported cstate change: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b92ff49cec804b1f801c11b64e3259b3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:55.579106 673 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:55.761075 31584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a5406bce7014bc2a7a1d727cb4aa191"
dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41212
I20240503 02:13:55.761343 31584 raft_consensus.cc:595] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:55.951640 4886 raft_consensus.cc:979] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3: : Instructing follower b52c8be37a2241ff9dc555aa0cebd5e5 to start an election
I20240503 02:13:55.951843 4886 raft_consensus.cc:1067] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Signalling peer b52c8be37a2241ff9dc555aa0cebd5e5 to start an election
I20240503 02:13:55.952450 31191 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a5406bce7014bc2a7a1d727cb4aa191"
dest_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5"
from {username='slave'} at 127.23.104.196:48301
I20240503 02:13:55.952653 31191 raft_consensus.cc:482] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:55.952766 31191 raft_consensus.cc:3037] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:55.955130 31191 raft_consensus.cc:504] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:55.955982 31191 leader_election.cc:288] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 2 election: Requested vote from peers b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849), 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543), 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997)
I20240503 02:13:55.956275 31584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5406bce7014bc2a7a1d727cb4aa191" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b92ff49cec804b1f801c11b64e3259b3"
I20240503 02:13:55.956337 673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5406bce7014bc2a7a1d727cb4aa191" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "13d0f62249b34b01a3a47c6afe10e750"
I20240503 02:13:55.956491 31584 raft_consensus.cc:3032] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:55.956561 673 raft_consensus.cc:3037] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:55.956574 31584 raft_consensus.cc:729] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b92ff49cec804b1f801c11b64e3259b3, State: Running, Role: LEADER
I20240503 02:13:55.956542 936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5406bce7014bc2a7a1d727cb4aa191" candidate_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4a3e3937b61447598fd76bc57f5d0939"
I20240503 02:13:55.956739 936 raft_consensus.cc:3037] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:55.956866 31584 consensus_queue.cc:260] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:55.957331 31584 raft_consensus.cc:3037] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:55.958560 673 raft_consensus.cc:2454] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 2.
I20240503 02:13:55.959621 936 raft_consensus.cc:2454] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 2.
I20240503 02:13:55.960029 31584 raft_consensus.cc:2454] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c8be37a2241ff9dc555aa0cebd5e5 in term 2.
I20240503 02:13:55.960047 31127 leader_election.cc:302] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 13d0f62249b34b01a3a47c6afe10e750, 4a3e3937b61447598fd76bc57f5d0939, b52c8be37a2241ff9dc555aa0cebd5e5; no voters:
I20240503 02:13:55.960981 4864 raft_consensus.cc:2781] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:55.961382 4864 raft_consensus.cc:686] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 LEADER]: Becoming Leader. State: Replica: b52c8be37a2241ff9dc555aa0cebd5e5, State: Running, Role: LEADER
I20240503 02:13:55.961709 4864 consensus_queue.cc:237] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:55.971043 31044 catalog_manager.cc:5484] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: term changed from 1 to 2, leader changed from b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) to b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193). New cstate: current_term: 2 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:56.339978 936 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:56.340576 4864 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:56.414664 31584 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:56.415306 4864 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92ff49cec804b1f801c11b64e3259b3" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 39849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:56.420102 671 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:56.420706 4864 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:56.954696 31191 consensus_queue.cc:237] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:13:56.956341 671 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:56.956391 936 raft_consensus.cc:1261] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Refusing update from remote peer b52c8be37a2241ff9dc555aa0cebd5e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:56.956826 5000 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:56.957096 4999 consensus_queue.cc:1035] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:56.959239 4981 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:56.959906 671 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P 13d0f62249b34b01a3a47c6afe10e750 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:56.962762 935 raft_consensus.cc:2932] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } }
I20240503 02:13:56.965236 31048 catalog_manager.cc:5484] T 8a5406bce7014bc2a7a1d727cb4aa191 P b52c8be37a2241ff9dc555aa0cebd5e5 reported cstate change: config changed from index 8 to 10, VOTER b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196) evicted. New cstate: current_term: 2 leader_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:56.967964 31038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a5406bce7014bc2a7a1d727cb4aa191 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:56.973829 31563 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a5406bce7014bc2a7a1d727cb4aa191 with delete_type TABLET_DATA_TOMBSTONED (TS b92ff49cec804b1f801c11b64e3259b3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36462
I20240503 02:13:56.975953 5006 tablet_replica.cc:330] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3: stopping tablet replica
I20240503 02:13:56.976219 5006 raft_consensus.cc:2229] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:56.976481 5006 raft_consensus.cc:2258] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:56.977494 5006 ts_tablet_manager.cc:1875] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:56.982805 5006 ts_tablet_manager.cc:1888] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 02:13:56.982929 5006 log.cc:1196] T 8a5406bce7014bc2a7a1d727cb4aa191 P b92ff49cec804b1f801c11b64e3259b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/8a5406bce7014bc2a7a1d727cb4aa191
I20240503 02:13:56.983687 31039 catalog_manager.cc:4830] TS b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849): tablet 8a5406bce7014bc2a7a1d727cb4aa191 (table rebalance_test_table_2 [id=88d4dd05cfa84299b1e62ec2a8b6a517]) successfully deleted
I20240503 02:13:57.143431 784 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:57.144421 1048 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:57.176071 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:57.183415 652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:57.216193 1951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:57.217387 31171 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:57.244505 31563 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:57.299392 31433 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:57.321367 31303 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:57.490325 1048 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:57.497627 784 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:57.516541 31563 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:57.518122 652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:57.525606 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:57.528096 31171 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:57.548517 31303 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:57.556476 31433 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:57.581555 1951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:58.456494 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:58.470001 784 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:58.471540 1048 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:58.485929 652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:58.490046 31171 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:58.535037 1951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:58.550205 31563 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:58.573431 31303 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:58.598134 31433 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:58.789526 1048 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:58.795459 784 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:58.811945 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:58.812494 652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:58.826738 31171 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:58.849136 31303 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:58.876232 31563 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:58.880293 1951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:58.900985 31433 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:59.138041 1048 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:59.147053 784 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:59.152288 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:59.173758 1951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:59.197849 652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:59.203217 31171 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:59.222574 31563 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:59.234344 31303 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:59.248075 31433 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:59.570827 1048 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:59.577450 784 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:59.599457 652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:59.601230 31171 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:59.612583 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:59.621006 1951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:59.644769 31563 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:59.658071 31303 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:59.675410 31433 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
32ac8e5f9d824ced889084d36419cdf5 | 127.23.104.254:43173 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.23.104.212:41411 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
0429a57df6ba4f92a73e779b04b05040 | 127.23.104.200:37401 | HEALTHY | <none> | 1 | 0
05ef0a15decc468ea3a1af8828b0f630 | 127.23.104.198:38195 | HEALTHY | <none> | 1 | 0
13d0f62249b34b01a3a47c6afe10e750 | 127.23.104.197:37543 | HEALTHY | <none> | 0 | 0
4a3e3937b61447598fd76bc57f5d0939 | 127.23.104.199:36997 | HEALTHY | <none> | 0 | 0
6ad20578b4a74bfabb6d1404de034a65 | 127.23.104.201:40601 | HEALTHY | <none> | 0 | 0
b52c8be37a2241ff9dc555aa0cebd5e5 | 127.23.104.193:44019 | HEALTHY | <none> | 6 | 0
b92ff49cec804b1f801c11b64e3259b3 | 127.23.104.196:39849 | HEALTHY | <none> | 6 | 0
cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 | 127.23.104.194:46179 | HEALTHY | <none> | 9 | 0
fbb7d16e67d94775b2881712419c0577 | 127.23.104.195:39101 | HEALTHY | <none> | 7 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 9
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
---------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 9 server(s) checked
local_ip_for_outbound_sockets | 127.23.104.193 | experimental | 127.23.104.193:44019
local_ip_for_outbound_sockets | 127.23.104.194 | experimental | 127.23.104.194:46179
local_ip_for_outbound_sockets | 127.23.104.195 | experimental | 127.23.104.195:39101
local_ip_for_outbound_sockets | 127.23.104.196 | experimental | 127.23.104.196:39849
local_ip_for_outbound_sockets | 127.23.104.197 | experimental | 127.23.104.197:37543
local_ip_for_outbound_sockets | 127.23.104.198 | experimental | 127.23.104.198:38195
local_ip_for_outbound_sockets | 127.23.104.199 | experimental | 127.23.104.199:36997
local_ip_for_outbound_sockets | 127.23.104.200 | experimental | 127.23.104.200:37401
local_ip_for_outbound_sockets | 127.23.104.201 | experimental | 127.23.104.201:40601
never_fsync | true | unsafe,advanced | all 9 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 9 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 9 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 9 server(s) checked
server_dump_info_format | pb | hidden | all 9 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.104.193:44019
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.23.104.194:46179
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.104.195:39101
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.23.104.196:39849
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.23.104.197:37543
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.23.104.198:38195
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.23.104.199:36997
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.23.104.200:37401
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data/info.pb | hidden | 127.23.104.201:40601
tablet_copy_download_file_inject_latency_ms | 1500 | hidden | all 9 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.23.104.212:41411 | all 9 server(s) checked
time_source | builtin | all 9 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------
1.18.0-SNAPSHOT | all 10 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 9
Tables | 10
Tablets | 30
Replicas | 90
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 02:13:59.758633 23971 log_verifier.cc:126] Checking tablet 00e1dad5efda4e9eae55a8f9d80f0f96
I20240503 02:13:59.810076 23971 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 00e1dad5efda4e9eae55a8f9d80f0f96
I20240503 02:13:59.810209 23971 log_verifier.cc:126] Checking tablet 091475e28bb344e78b1e4e48fa548ef8
I20240503 02:13:59.844632 23971 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 091475e28bb344e78b1e4e48fa548ef8
I20240503 02:13:59.844733 23971 log_verifier.cc:126] Checking tablet 0ce9e7d46a6e475faaffb1ee26973db7
I20240503 02:13:59.880646 23971 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 0ce9e7d46a6e475faaffb1ee26973db7
I20240503 02:13:59.880749 23971 log_verifier.cc:126] Checking tablet 17be048393974fdf9811c5b8f8a97579
I20240503 02:13:59.917112 23971 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 17be048393974fdf9811c5b8f8a97579
I20240503 02:13:59.917256 23971 log_verifier.cc:126] Checking tablet 1c425b03a1c64d7ba7a14901d63d7037
I20240503 02:13:59.992967 23971 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 1c425b03a1c64d7ba7a14901d63d7037
I20240503 02:13:59.993090 23971 log_verifier.cc:126] Checking tablet 2de731c4261f49088958535b8b7a65e9
I20240503 02:14:00.034417 23971 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 2de731c4261f49088958535b8b7a65e9
I20240503 02:14:00.034513 23971 log_verifier.cc:126] Checking tablet 37ca6554fec24a07865da3eabef27152
I20240503 02:14:00.070052 23971 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 37ca6554fec24a07865da3eabef27152
I20240503 02:14:00.070156 23971 log_verifier.cc:126] Checking tablet 3f8353133ccd4601a3761652546a33b5
I20240503 02:14:00.115377 23971 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 3f8353133ccd4601a3761652546a33b5
I20240503 02:14:00.115530 23971 log_verifier.cc:126] Checking tablet 4995796bdda64a7cbffa952a05a36673
I20240503 02:14:00.157407 23971 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 4995796bdda64a7cbffa952a05a36673
I20240503 02:14:00.157505 23971 log_verifier.cc:126] Checking tablet 4dc576d8e72647c08736ae9508f6d27b
I20240503 02:14:00.193874 23971 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 4dc576d8e72647c08736ae9508f6d27b
I20240503 02:14:00.193972 23971 log_verifier.cc:126] Checking tablet 6c3d978cd00449b8b3edd4c43b657960
I20240503 02:14:00.263592 23971 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 6c3d978cd00449b8b3edd4c43b657960
I20240503 02:14:00.263701 23971 log_verifier.cc:126] Checking tablet 6ce56ca4030a445db890303244caf308
I20240503 02:14:00.302639 23971 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 6ce56ca4030a445db890303244caf308
I20240503 02:14:00.302764 23971 log_verifier.cc:126] Checking tablet 6f8e7774779e47ba9d426ac00ad8be62
I20240503 02:14:00.338760 23971 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 6f8e7774779e47ba9d426ac00ad8be62
I20240503 02:14:00.338872 23971 log_verifier.cc:126] Checking tablet 7230f1d648c34351aed29f98795150a2
I20240503 02:14:00.377457 23971 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 7230f1d648c34351aed29f98795150a2
I20240503 02:14:00.377581 23971 log_verifier.cc:126] Checking tablet 7626c06dc67448f0b65f67b401d62c88
I20240503 02:14:00.415143 23971 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 7626c06dc67448f0b65f67b401d62c88
I20240503 02:14:00.415264 23971 log_verifier.cc:126] Checking tablet 80a15c3541fc4068acac595bd909f8d8
I20240503 02:14:00.454393 23971 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 80a15c3541fc4068acac595bd909f8d8
I20240503 02:14:00.454509 23971 log_verifier.cc:126] Checking tablet 8a5406bce7014bc2a7a1d727cb4aa191
I20240503 02:14:00.521667 23971 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 8a5406bce7014bc2a7a1d727cb4aa191
I20240503 02:14:00.521766 23971 log_verifier.cc:126] Checking tablet 8c681bdcf3074c54b1d533cf1f7d2b75
I20240503 02:14:00.566893 23971 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 8c681bdcf3074c54b1d533cf1f7d2b75
I20240503 02:14:00.566998 23971 log_verifier.cc:126] Checking tablet 8d8d17ee831446ae8d4a0417ede894b8
I20240503 02:14:00.605726 23971 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 8d8d17ee831446ae8d4a0417ede894b8
I20240503 02:14:00.605840 23971 log_verifier.cc:126] Checking tablet 9000ce76eb154434a88b52e357497767
I20240503 02:14:00.645296 23971 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 9000ce76eb154434a88b52e357497767
I20240503 02:14:00.645414 23971 log_verifier.cc:126] Checking tablet 9b909a66859144f7b4b649d505c9645e
I20240503 02:14:00.683107 23971 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 9b909a66859144f7b4b649d505c9645e
I20240503 02:14:00.683229 23971 log_verifier.cc:126] Checking tablet 9e964b80f5414d9ea3a5aea07b99d1d4
I20240503 02:14:00.721879 23971 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 9e964b80f5414d9ea3a5aea07b99d1d4
I20240503 02:14:00.721998 23971 log_verifier.cc:126] Checking tablet b6c71c82cf034bd68aa725e086c56ffb
I20240503 02:14:00.786862 23971 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b6c71c82cf034bd68aa725e086c56ffb
I20240503 02:14:00.786957 23971 log_verifier.cc:126] Checking tablet b76fa26b48e942ed840b5ee55ce40387
I20240503 02:14:00.825661 23971 log_verifier.cc:177] Verified matching terms for 9 ops in tablet b76fa26b48e942ed840b5ee55ce40387
I20240503 02:14:00.825760 23971 log_verifier.cc:126] Checking tablet c185751912de4f78808785a620dd9746
I20240503 02:14:00.861127 23971 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c185751912de4f78808785a620dd9746
I20240503 02:14:00.861239 23971 log_verifier.cc:126] Checking tablet c46912d2d4434d4a8c3fa0c3c19ba14b
I20240503 02:14:00.896553 23971 log_verifier.cc:177] Verified matching terms for 10 ops in tablet c46912d2d4434d4a8c3fa0c3c19ba14b
I20240503 02:14:00.896673 23971 log_verifier.cc:126] Checking tablet c91a7b003ed9411aba65a46d00c7d95b
I20240503 02:14:00.935899 23971 log_verifier.cc:177] Verified matching terms for 10 ops in tablet c91a7b003ed9411aba65a46d00c7d95b
I20240503 02:14:00.936026 23971 log_verifier.cc:126] Checking tablet cb7e8fe39c024108be24bc4c144b247b
I20240503 02:14:00.978677 23971 log_verifier.cc:177] Verified matching terms for 9 ops in tablet cb7e8fe39c024108be24bc4c144b247b
I20240503 02:14:00.978809 23971 log_verifier.cc:126] Checking tablet d9df1587246a4df19fda13f4be2b1177
I20240503 02:14:01.050343 23971 log_verifier.cc:177] Verified matching terms for 9 ops in tablet d9df1587246a4df19fda13f4be2b1177
I20240503 02:14:01.050441 23971 log_verifier.cc:126] Checking tablet f52b9a3c14a04815800753bda9505cee
I20240503 02:14:01.085144 23971 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f52b9a3c14a04815800753bda9505cee
I20240503 02:14:01.086028 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 31111
I20240503 02:14:01.098285 31233 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:01.302865 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 31111
W20240503 02:14:01.336761 1003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.104.193:44019 (error 108)
W20240503 02:14:01.336872 31520 connection.cc:535] server connection from 127.23.104.193:59475 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:01.337471 31519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.104.193:44019 (error 108)
W20240503 02:14:01.337831 1004 connection.cc:535] server connection from 127.23.104.193:56665 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:14:01.338423 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 31242
W20240503 02:14:01.339608 1003 consensus_peers.cc:485] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 -> Peer b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): Couldn't send request to peer b52c8be37a2241ff9dc555aa0cebd5e5. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:44019: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:01.339637 31519 consensus_peers.cc:485] T 091475e28bb344e78b1e4e48fa548ef8 P b92ff49cec804b1f801c11b64e3259b3 -> Peer b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): Couldn't send request to peer b52c8be37a2241ff9dc555aa0cebd5e5. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:44019: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:01.344372 31363 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:14:01.489846 31389 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.104.193:44019: connect: Connection refused (error 111)
W20240503 02:14:01.491137 31389 consensus_peers.cc:485] T 6f8e7774779e47ba9d426ac00ad8be62 P fbb7d16e67d94775b2881712419c0577 -> Peer b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): Couldn't send request to peer b52c8be37a2241ff9dc555aa0cebd5e5. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:44019: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:01.537401 31519 consensus_peers.cc:485] T 2de731c4261f49088958535b8b7a65e9 P b92ff49cec804b1f801c11b64e3259b3 -> Peer b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): Couldn't send request to peer b52c8be37a2241ff9dc555aa0cebd5e5. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:44019: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:01.590610 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 31242
W20240503 02:14:01.628304 31388 connection.cc:535] client connection to 127.23.104.194:46179 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:14:01.628789 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 31372
W20240503 02:14:01.628952 1001 connection.cc:535] server connection from 127.23.104.194:52941 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:01.629258 737 connection.cc:535] server connection from 127.23.104.194:50489 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:01.629926 31520 connection.cc:535] server connection from 127.23.104.194:49625 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:01.630483 31388 consensus_peers.cc:485] T 37ca6554fec24a07865da3eabef27152 P fbb7d16e67d94775b2881712419c0577 -> Peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): Couldn't send request to peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.194:46179: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:01.635848 31494 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:01.856243 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 31372
W20240503 02:14:01.897559 739 connection.cc:535] server connection from 127.23.104.195:36249 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:01.897989 870 connection.cc:535] server connection from 127.23.104.195:49803 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:01.898309 1860 connection.cc:535] server connection from 127.23.104.195:56115 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:01.898314 1004 connection.cc:535] server connection from 127.23.104.195:37547 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:14:01.898370 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 31502
W20240503 02:14:01.899667 31517 consensus_peers.cc:485] T c185751912de4f78808785a620dd9746 P b92ff49cec804b1f801c11b64e3259b3 -> Peer fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101): Couldn't send request to peer fbb7d16e67d94775b2881712419c0577. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.195:39101: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:01.904628 31622 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:02.109525 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 31502
W20240503 02:14:02.143976 1002 connection.cc:535] server connection from 127.23.104.196:57909 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:02.144019 869 connection.cc:535] server connection from 127.23.104.196:41577 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:02.144516 609 connection.cc:535] server connection from 127.23.104.196:50843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:14:02.144693 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 591
I20240503 02:14:02.150292 713 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:14:02.198493 740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.104.196:39849: connect: Connection refused (error 111)
W20240503 02:14:02.199980 740 consensus_peers.cc:485] T 6ce56ca4030a445db890303244caf308 P 05ef0a15decc468ea3a1af8828b0f630 -> Peer b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849): Couldn't send request to peer b92ff49cec804b1f801c11b64e3259b3. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.196:39849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:02.296312 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 591
I20240503 02:14:02.321813 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 722
I20240503 02:14:02.327538 842 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:02.364534 5308 raft_consensus.cc:482] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b52c8be37a2241ff9dc555aa0cebd5e5)
I20240503 02:14:02.364773 5308 raft_consensus.cc:504] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.365773 5308 leader_election.cc:288] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601)
W20240503 02:14:02.367806 1003 leader_election.cc:334] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): Network error: Client connection negotiation failed: client connection to 127.23.104.193:44019: connect: Connection refused (error 111)
I20240503 02:14:02.372539 1970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e1dad5efda4e9eae55a8f9d80f0f96" candidate_uuid: "0429a57df6ba4f92a73e779b04b05040" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "6ad20578b4a74bfabb6d1404de034a65" is_pre_election: true
I20240503 02:14:02.373229 1001 leader_election.cc:302] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0429a57df6ba4f92a73e779b04b05040; no voters: 6ad20578b4a74bfabb6d1404de034a65, b52c8be37a2241ff9dc555aa0cebd5e5
I20240503 02:14:02.373531 5308 raft_consensus.cc:2726] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:14:02.393895 1004 consensus_peers.cc:485] T 4dc576d8e72647c08736ae9508f6d27b P 0429a57df6ba4f92a73e779b04b05040 -> Peer b92ff49cec804b1f801c11b64e3259b3 (127.23.104.196:39849): Couldn't send request to peer b92ff49cec804b1f801c11b64e3259b3. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.196:39849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:02.408692 5330 raft_consensus.cc:482] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd37b5f7f6ff4f4ca4105ad5ecf9c3c3)
I20240503 02:14:02.408973 5330 raft_consensus.cc:504] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.410480 5330 leader_election.cc:288] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401)
W20240503 02:14:02.428808 1860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.104.194:46179: connect: Connection refused (error 111)
I20240503 02:14:02.432629 5334 raft_consensus.cc:482] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b52c8be37a2241ff9dc555aa0cebd5e5)
I20240503 02:14:02.432883 5334 raft_consensus.cc:504] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:14:02.437521 5334 leader_election.cc:288] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543)
I20240503 02:14:02.437575 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76fa26b48e942ed840b5ee55ce40387" candidate_uuid: "6ad20578b4a74bfabb6d1404de034a65" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "0429a57df6ba4f92a73e779b04b05040" is_pre_election: true
W20240503 02:14:02.439450 871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.104.193:44019: connect: Connection refused (error 111)
W20240503 02:14:02.439553 1860 leader_election.cc:334] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): Network error: Client connection negotiation failed: client connection to 127.23.104.194:46179: connect: Connection refused (error 111)
I20240503 02:14:02.439719 1860 leader_election.cc:302] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ad20578b4a74bfabb6d1404de034a65; no voters: 0429a57df6ba4f92a73e779b04b05040, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3
I20240503 02:14:02.440044 5330 raft_consensus.cc:2726] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:14:02.441358 870 leader_election.cc:334] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543): Network error: Client connection negotiation failed: client connection to 127.23.104.197:37543: connect: Connection refused (error 111)
W20240503 02:14:02.441532 871 leader_election.cc:334] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): Network error: Client connection negotiation failed: client connection to 127.23.104.193:44019: connect: Connection refused (error 111)
I20240503 02:14:02.441677 871 leader_election.cc:302] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a3e3937b61447598fd76bc57f5d0939; no voters: 13d0f62249b34b01a3a47c6afe10e750, b52c8be37a2241ff9dc555aa0cebd5e5
I20240503 02:14:02.441988 5334 raft_consensus.cc:2726] T 8a5406bce7014bc2a7a1d727cb4aa191 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:14:02.467566 5308 raft_consensus.cc:482] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b52c8be37a2241ff9dc555aa0cebd5e5)
I20240503 02:14:02.467775 5308 raft_consensus.cc:504] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } }
I20240503 02:14:02.468143 5334 raft_consensus.cc:482] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd37b5f7f6ff4f4ca4105ad5ecf9c3c3)
I20240503 02:14:02.468310 5334 raft_consensus.cc:504] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.468648 5308 leader_election.cc:288] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997)
I20240503 02:14:02.470278 5334 leader_election.cc:288] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601)
W20240503 02:14:02.470628 1003 leader_election.cc:334] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): Network error: Client connection negotiation failed: client connection to 127.23.104.193:44019: connect: Connection refused (error 111)
W20240503 02:14:02.471967 870 leader_election.cc:334] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): Network error: Client connection negotiation failed: client connection to 127.23.104.194:46179: connect: Connection refused (error 111)
I20240503 02:14:02.476423 934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4995796bdda64a7cbffa952a05a36673" candidate_uuid: "0429a57df6ba4f92a73e779b04b05040" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "4a3e3937b61447598fd76bc57f5d0939" is_pre_election: true
I20240503 02:14:02.476725 1970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17be048393974fdf9811c5b8f8a97579" candidate_uuid: "4a3e3937b61447598fd76bc57f5d0939" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "6ad20578b4a74bfabb6d1404de034a65" is_pre_election: true
I20240503 02:14:02.477113 1002 leader_election.cc:302] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0429a57df6ba4f92a73e779b04b05040; no voters: 4a3e3937b61447598fd76bc57f5d0939, b52c8be37a2241ff9dc555aa0cebd5e5
I20240503 02:14:02.477279 869 leader_election.cc:302] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a3e3937b61447598fd76bc57f5d0939; no voters: 6ad20578b4a74bfabb6d1404de034a65, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3
I20240503 02:14:02.477396 5308 raft_consensus.cc:2726] T 4995796bdda64a7cbffa952a05a36673 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:02.477591 5334 raft_consensus.cc:2726] T 17be048393974fdf9811c5b8f8a97579 P 4a3e3937b61447598fd76bc57f5d0939 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:14:02.480538 5330 raft_consensus.cc:482] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd37b5f7f6ff4f4ca4105ad5ecf9c3c3)
I20240503 02:14:02.480706 5330 raft_consensus.cc:504] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.481523 5330 leader_election.cc:288] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195)
I20240503 02:14:02.481670 5330 raft_consensus.cc:482] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd37b5f7f6ff4f4ca4105ad5ecf9c3c3)
I20240503 02:14:02.481877 5330 raft_consensus.cc:504] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.482518 5330 leader_election.cc:288] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401)
I20240503 02:14:02.483012 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7e8fe39c024108be24bc4c144b247b" candidate_uuid: "6ad20578b4a74bfabb6d1404de034a65" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "0429a57df6ba4f92a73e779b04b05040" is_pre_election: true
W20240503 02:14:02.484251 1860 leader_election.cc:334] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): Network error: Client connection negotiation failed: client connection to 127.23.104.194:46179: connect: Connection refused (error 111)
W20240503 02:14:02.484396 1860 leader_election.cc:334] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): Network error: Client connection negotiation failed: client connection to 127.23.104.194:46179: connect: Connection refused (error 111)
I20240503 02:14:02.484501 1860 leader_election.cc:302] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ad20578b4a74bfabb6d1404de034a65; no voters: 0429a57df6ba4f92a73e779b04b05040, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3
I20240503 02:14:02.484750 5340 raft_consensus.cc:2726] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:02.486791 5340 raft_consensus.cc:482] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b52c8be37a2241ff9dc555aa0cebd5e5)
I20240503 02:14:02.486965 5340 raft_consensus.cc:504] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.487730 5340 leader_election.cc:288] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401)
I20240503 02:14:02.488234 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e1dad5efda4e9eae55a8f9d80f0f96" candidate_uuid: "6ad20578b4a74bfabb6d1404de034a65" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "0429a57df6ba4f92a73e779b04b05040" is_pre_election: true
I20240503 02:14:02.488402 1066 raft_consensus.cc:2454] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad20578b4a74bfabb6d1404de034a65 in term 2.
I20240503 02:14:02.488776 1862 leader_election.cc:302] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0429a57df6ba4f92a73e779b04b05040, 6ad20578b4a74bfabb6d1404de034a65; no voters:
I20240503 02:14:02.489080 5340 raft_consensus.cc:2781] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:14:02.489187 5340 raft_consensus.cc:482] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b52c8be37a2241ff9dc555aa0cebd5e5)
I20240503 02:14:02.489300 5340 raft_consensus.cc:3037] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:14:02.489446 1861 leader_election.cc:334] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): Network error: Client connection negotiation failed: client connection to 127.23.104.193:44019: connect: Connection refused (error 111)
I20240503 02:14:02.490792 5308 raft_consensus.cc:482] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b52c8be37a2241ff9dc555aa0cebd5e5)
I20240503 02:14:02.490949 5308 raft_consensus.cc:504] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:14:02.491770 5308 leader_election.cc:288] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101)
I20240503 02:14:02.491683 5340 raft_consensus.cc:504] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.492410 5340 leader_election.cc:288] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 3 election: Requested vote from peers b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019), 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200:37401)
I20240503 02:14:02.492717 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e1dad5efda4e9eae55a8f9d80f0f96" candidate_uuid: "6ad20578b4a74bfabb6d1404de034a65" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "0429a57df6ba4f92a73e779b04b05040"
I20240503 02:14:02.492890 1066 raft_consensus.cc:3037] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:14:02.493376 1003 leader_election.cc:334] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): Network error: Client connection negotiation failed: client connection to 127.23.104.193:44019: connect: Connection refused (error 111)
W20240503 02:14:02.493825 1001 leader_election.cc:334] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fbb7d16e67d94775b2881712419c0577 (127.23.104.195:39101): Network error: Client connection negotiation failed: client connection to 127.23.104.195:39101: connect: Connection refused (error 111)
I20240503 02:14:02.493952 1001 leader_election.cc:302] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0429a57df6ba4f92a73e779b04b05040; no voters: b52c8be37a2241ff9dc555aa0cebd5e5, fbb7d16e67d94775b2881712419c0577
I20240503 02:14:02.494488 5308 raft_consensus.cc:2726] T 0ce9e7d46a6e475faaffb1ee26973db7 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:02.496315 1066 raft_consensus.cc:2454] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 0429a57df6ba4f92a73e779b04b05040 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad20578b4a74bfabb6d1404de034a65 in term 3.
W20240503 02:14:02.496676 1861 leader_election.cc:334] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193:44019): Network error: Client connection negotiation failed: client connection to 127.23.104.193:44019: connect: Connection refused (error 111)
I20240503 02:14:02.497100 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 722
I20240503 02:14:02.497206 1862 leader_election.cc:302] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0429a57df6ba4f92a73e779b04b05040, 6ad20578b4a74bfabb6d1404de034a65; no voters: b52c8be37a2241ff9dc555aa0cebd5e5
I20240503 02:14:02.498138 5340 raft_consensus.cc:2781] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:14:02.498292 5340 raft_consensus.cc:686] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [term 3 LEADER]: Becoming Leader. State: Replica: 6ad20578b4a74bfabb6d1404de034a65, State: Running, Role: LEADER
I20240503 02:14:02.498745 5340 consensus_queue.cc:237] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.502440 31048 catalog_manager.cc:5484] T 00e1dad5efda4e9eae55a8f9d80f0f96 P 6ad20578b4a74bfabb6d1404de034a65 reported cstate change: term changed from 2 to 3, leader changed from b52c8be37a2241ff9dc555aa0cebd5e5 (127.23.104.193) to 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201). New cstate: current_term: 3 leader_uuid: "6ad20578b4a74bfabb6d1404de034a65" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 02:14:02.524078 870 connection.cc:535] server connection from 127.23.104.198:59731 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:14:02.524511 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 852
W20240503 02:14:02.525476 5332 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:14:02.481819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.104.198:38195 (local address 127.23.104.201:45931)
0503 02:14:02.482103 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0503 02:14:02.482123 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0503 02:14:02.482255 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:14:02.496262 (+ 14007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 02:14:02.496269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 02:14:02.496301 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 02:14:02.496801 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 02:14:02.496816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 02:14:02.524606 (+ 27790us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.104.198:38195: BlockingRecv error: recv got EOF from 127.23.104.198:38195 (error 108)
Metrics: {"client-negotiator.queue_time_us":169}
W20240503 02:14:02.526475 1861 leader_election.cc:334] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195): Network error: Client connection negotiation failed: client connection to 127.23.104.198:38195: connect: Connection refused (error 111)
I20240503 02:14:02.526614 1861 leader_election.cc:302] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ad20578b4a74bfabb6d1404de034a65; no voters: 05ef0a15decc468ea3a1af8828b0f630, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3
I20240503 02:14:02.526901 5340 raft_consensus.cc:2726] T 9e964b80f5414d9ea3a5aea07b99d1d4 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:02.530332 976 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:02.544484 5340 raft_consensus.cc:482] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd37b5f7f6ff4f4ca4105ad5ecf9c3c3)
I20240503 02:14:02.544693 5340 raft_consensus.cc:504] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.545372 5340 leader_election.cc:288] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543)
W20240503 02:14:02.546906 1860 leader_election.cc:334] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): Network error: Client connection negotiation failed: client connection to 127.23.104.194:46179: connect: Connection refused (error 111)
W20240503 02:14:02.547133 1860 leader_election.cc:334] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 13d0f62249b34b01a3a47c6afe10e750 (127.23.104.197:37543): Network error: Client connection negotiation failed: client connection to 127.23.104.197:37543: connect: Connection refused (error 111)
I20240503 02:14:02.547225 1860 leader_election.cc:302] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ad20578b4a74bfabb6d1404de034a65; no voters: 13d0f62249b34b01a3a47c6afe10e750, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3
I20240503 02:14:02.547485 5340 raft_consensus.cc:2726] T 9b909a66859144f7b4b649d505c9645e P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:02.558883 5340 raft_consensus.cc:482] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd37b5f7f6ff4f4ca4105ad5ecf9c3c3)
I20240503 02:14:02.559093 5340 raft_consensus.cc:504] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "4a3e3937b61447598fd76bc57f5d0939" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.559760 5340 leader_election.cc:288] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997)
W20240503 02:14:02.561220 1860 leader_election.cc:334] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): Network error: Client connection negotiation failed: client connection to 127.23.104.194:46179: connect: Connection refused (error 111)
I20240503 02:14:02.618570 5308 raft_consensus.cc:482] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd37b5f7f6ff4f4ca4105ad5ecf9c3c3)
I20240503 02:14:02.618763 5308 raft_consensus.cc:504] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.619563 5308 leader_election.cc:288] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601)
I20240503 02:14:02.620168 1970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7e8fe39c024108be24bc4c144b247b" candidate_uuid: "0429a57df6ba4f92a73e779b04b05040" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "6ad20578b4a74bfabb6d1404de034a65" is_pre_election: true
I20240503 02:14:02.620442 1970 raft_consensus.cc:2454] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0429a57df6ba4f92a73e779b04b05040 in term 1.
I20240503 02:14:02.620827 1001 leader_election.cc:302] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0429a57df6ba4f92a73e779b04b05040, 6ad20578b4a74bfabb6d1404de034a65; no voters:
W20240503 02:14:02.620991 1002 leader_election.cc:334] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): Network error: Client connection negotiation failed: client connection to 127.23.104.194:46179: connect: Connection refused (error 111)
I20240503 02:14:02.621126 5308 raft_consensus.cc:2781] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:14:02.621249 5308 raft_consensus.cc:482] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd37b5f7f6ff4f4ca4105ad5ecf9c3c3)
I20240503 02:14:02.621351 5308 raft_consensus.cc:3037] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:02.624295 5308 raft_consensus.cc:504] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.624946 5308 leader_election.cc:288] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 election: Requested vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601)
I20240503 02:14:02.625414 1970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7e8fe39c024108be24bc4c144b247b" candidate_uuid: "0429a57df6ba4f92a73e779b04b05040" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "6ad20578b4a74bfabb6d1404de034a65"
I20240503 02:14:02.625640 1970 raft_consensus.cc:3037] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:14:02.626155 1002 leader_election.cc:334] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): Network error: Client connection negotiation failed: client connection to 127.23.104.194:46179: connect: Connection refused (error 111)
I20240503 02:14:02.627748 1970 raft_consensus.cc:2454] T cb7e8fe39c024108be24bc4c144b247b P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0429a57df6ba4f92a73e779b04b05040 in term 2.
I20240503 02:14:02.628150 1001 leader_election.cc:302] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0429a57df6ba4f92a73e779b04b05040, 6ad20578b4a74bfabb6d1404de034a65; no voters: cd37b5f7f6ff4f4ca4105ad5ecf9c3c3
I20240503 02:14:02.628412 5308 raft_consensus.cc:2781] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:02.628863 5308 raft_consensus.cc:686] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [term 2 LEADER]: Becoming Leader. State: Replica: 0429a57df6ba4f92a73e779b04b05040, State: Running, Role: LEADER
I20240503 02:14:02.629192 5308 consensus_queue.cc:237] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.632356 31048 catalog_manager.cc:5484] T cb7e8fe39c024108be24bc4c144b247b P 0429a57df6ba4f92a73e779b04b05040 reported cstate change: term changed from 1 to 2, leader changed from cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) to 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200). New cstate: current_term: 2 leader_uuid: "0429a57df6ba4f92a73e779b04b05040" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:02.678689 5308 raft_consensus.cc:482] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd37b5f7f6ff4f4ca4105ad5ecf9c3c3)
I20240503 02:14:02.678943 5308 raft_consensus.cc:504] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.679740 5308 leader_election.cc:288] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601)
I20240503 02:14:02.680464 1970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76fa26b48e942ed840b5ee55ce40387" candidate_uuid: "0429a57df6ba4f92a73e779b04b05040" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "6ad20578b4a74bfabb6d1404de034a65" is_pre_election: true
I20240503 02:14:02.680718 1970 raft_consensus.cc:2454] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0429a57df6ba4f92a73e779b04b05040 in term 1.
I20240503 02:14:02.681128 1001 leader_election.cc:302] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0429a57df6ba4f92a73e779b04b05040, 6ad20578b4a74bfabb6d1404de034a65; no voters:
W20240503 02:14:02.681320 1002 leader_election.cc:334] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): Network error: Client connection negotiation failed: client connection to 127.23.104.194:46179: connect: Connection refused (error 111)
I20240503 02:14:02.681416 5308 raft_consensus.cc:2781] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:14:02.681493 5308 raft_consensus.cc:482] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd37b5f7f6ff4f4ca4105ad5ecf9c3c3)
I20240503 02:14:02.681639 5308 raft_consensus.cc:3037] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:02.683511 5308 raft_consensus.cc:504] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.684167 5308 leader_election.cc:288] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 election: Requested vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), 6ad20578b4a74bfabb6d1404de034a65 (127.23.104.201:40601)
I20240503 02:14:02.684703 1970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76fa26b48e942ed840b5ee55ce40387" candidate_uuid: "0429a57df6ba4f92a73e779b04b05040" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "6ad20578b4a74bfabb6d1404de034a65"
I20240503 02:14:02.684890 1970 raft_consensus.cc:3037] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:02.686142 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 852
I20240503 02:14:02.687951 1970 raft_consensus.cc:2454] T b76fa26b48e942ed840b5ee55ce40387 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0429a57df6ba4f92a73e779b04b05040 in term 2.
I20240503 02:14:02.688347 1001 leader_election.cc:302] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0429a57df6ba4f92a73e779b04b05040, 6ad20578b4a74bfabb6d1404de034a65; no voters:
I20240503 02:14:02.688584 5308 raft_consensus.cc:2781] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:02.688724 5308 raft_consensus.cc:686] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [term 2 LEADER]: Becoming Leader. State: Replica: 0429a57df6ba4f92a73e779b04b05040, State: Running, Role: LEADER
I20240503 02:14:02.688957 5308 consensus_queue.cc:237] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.691486 31048 catalog_manager.cc:5484] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 reported cstate change: term changed from 1 to 2, leader changed from cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194) to 0429a57df6ba4f92a73e779b04b05040 (127.23.104.200). New cstate: current_term: 2 leader_uuid: "0429a57df6ba4f92a73e779b04b05040" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:14:02.696126 1002 leader_election.cc:334] T b76fa26b48e942ed840b5ee55ce40387 P 0429a57df6ba4f92a73e779b04b05040 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): Network error: Client connection negotiation failed: client connection to 127.23.104.194:46179: connect: Connection refused (error 111)
W20240503 02:14:02.713747 5332 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:14:02.560272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.104.199:36997 (local address 127.23.104.201:47167)
0503 02:14:02.560411 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0503 02:14:02.560427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0503 02:14:02.560539 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:14:02.713539 (+153000us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.104.199:36997: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":58}
I20240503 02:14:02.714354 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 984
W20240503 02:14:02.714820 1860 leader_election.cc:334] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a3e3937b61447598fd76bc57f5d0939 (127.23.104.199:36997): Network error: Client connection negotiation failed: client connection to 127.23.104.199:36997: connect: Connection refused (error 111)
I20240503 02:14:02.714993 1860 leader_election.cc:302] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ad20578b4a74bfabb6d1404de034a65; no voters: 4a3e3937b61447598fd76bc57f5d0939, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3
I20240503 02:14:02.715394 5340 raft_consensus.cc:2726] T 17be048393974fdf9811c5b8f8a97579 P 6ad20578b4a74bfabb6d1404de034a65 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:14:02.720257 1105 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:02.734841 5340 raft_consensus.cc:482] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd37b5f7f6ff4f4ca4105ad5ecf9c3c3)
I20240503 02:14:02.735076 5340 raft_consensus.cc:504] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd37b5f7f6ff4f4ca4105ad5ecf9c3c3" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 46179 } } peers { permanent_uuid: "05ef0a15decc468ea3a1af8828b0f630" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 38195 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:02.735744 5340 leader_election.cc:288] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179), 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195)
W20240503 02:14:02.737018 1861 leader_election.cc:334] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 05ef0a15decc468ea3a1af8828b0f630 (127.23.104.198:38195): Network error: Client connection negotiation failed: client connection to 127.23.104.198:38195: connect: Connection refused (error 111)
W20240503 02:14:02.737205 1860 leader_election.cc:334] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cd37b5f7f6ff4f4ca4105ad5ecf9c3c3 (127.23.104.194:46179): Network error: Client connection negotiation failed: client connection to 127.23.104.194:46179: connect: Connection refused (error 111)
I20240503 02:14:02.737365 1860 leader_election.cc:302] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ad20578b4a74bfabb6d1404de034a65; no voters: 05ef0a15decc468ea3a1af8828b0f630, cd37b5f7f6ff4f4ca4105ad5ecf9c3c3
I20240503 02:14:02.737648 5340 raft_consensus.cc:2726] T 7626c06dc67448f0b65f67b401d62c88 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:02.894691 5308 raft_consensus.cc:482] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b52c8be37a2241ff9dc555aa0cebd5e5)
I20240503 02:14:02.894914 5308 raft_consensus.cc:504] T c91a7b003ed9411aba65a46d00c7d95b P 0429a57df6ba4f92a73e779b04b05040 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b52c8be37a2241ff9dc555aa0cebd5e5" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 44019 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "0429a57df6ba4f92a73e779b04b05040" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 37401 } attrs { promote: false } }
I20240503 02:14:02.895226 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 984
I20240503 02:14:02.921241 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 1768
I20240503 02:14:02.926991 2012 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:03.055331 5330 raft_consensus.cc:482] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fbb7d16e67d94775b2881712419c0577)
I20240503 02:14:03.055519 5330 raft_consensus.cc:504] T d9df1587246a4df19fda13f4be2b1177 P 6ad20578b4a74bfabb6d1404de034a65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fbb7d16e67d94775b2881712419c0577" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 39101 } } peers { permanent_uuid: "13d0f62249b34b01a3a47c6afe10e750" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "6ad20578b4a74bfabb6d1404de034a65" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 40601 } attrs { promote: false } }
I20240503 02:14:03.055830 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 1768
I20240503 02:14:03.079061 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 31021
I20240503 02:14:03.080087 31082 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:03.276559 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 31021
2024-05-03T02:14:03Z chronyd exiting
[ OK ] TserverAddedDuringRebalancingTest.TserverStarts/1 (53903 ms)
[----------] 1 test from TserverAddedDuringRebalancingTest (53903 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5
I20240503 02:14:03.444198 23971 test_util.cc:268] Using random seed: 1175025617
I20240503 02:14:03.445781 23971 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:14:03.445838 23971 ts_itest-base.cc:113] --------------
I20240503 02:14:03.445884 23971 ts_itest-base.cc:114] 4 tablet servers
I20240503 02:14:03.445935 23971 ts_itest-base.cc:115] 1 replicas per TS
I20240503 02:14:03.446004 23971 ts_itest-base.cc:116] --------------
2024-05-03T02:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:14:03Z Disabled control of system clock
I20240503 02:14:03.493862 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.104.254:43347
--webserver_interface=127.23.104.254
--webserver_port=0
--builtin_ntp_servers=127.23.104.212:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.104.254:43347
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240503 02:14:03.813596 5361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:03.813863 5361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:03.813941 5361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:03.820645 5361 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:03.820741 5361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:14:03.820784 5361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:03.820828 5361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:14:03.820869 5361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:14:03.829928 5361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.23.104.254:43347
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.104.254:43347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.23.104.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:03.831044 5361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:03.832625 5361 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:03.842720 5366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:03.843380 5367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:03.843955 5369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:03.844604 5361 server_base.cc:1034] running on GCE node
I20240503 02:14:03.845238 5361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:03.846580 5361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:03.847764 5361 hybrid_clock.cc:648] HybridClock initialized: now 1714702443847724 us; error 39 us; skew 500 ppm
I20240503 02:14:03.848109 5361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:03.850201 5361 webserver.cc:460] Webserver started at http://127.23.104.254:34767/ using document root <none> and password file <none>
I20240503 02:14:03.850859 5361 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:03.850975 5361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:03.851262 5361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:03.853816 5361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "385a520856d24cbab1558ef77c93b8c6"
format_stamp: "Formatted at 2024-05-03 02:14:03 on dist-test-slave-38ht"
I20240503 02:14:03.854559 5361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "385a520856d24cbab1558ef77c93b8c6"
format_stamp: "Formatted at 2024-05-03 02:14:03 on dist-test-slave-38ht"
I20240503 02:14:03.859184 5361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20240503 02:14:03.862215 5375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:03.863366 5361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:03.863516 5361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
uuid: "385a520856d24cbab1558ef77c93b8c6"
format_stamp: "Formatted at 2024-05-03 02:14:03 on dist-test-slave-38ht"
I20240503 02:14:03.863692 5361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:03.885601 5361 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:03.886307 5361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:03.886548 5361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:03.903642 5361 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.254:43347
I20240503 02:14:03.903671 5426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.254:43347 every 8 connection(s)
I20240503 02:14:03.904872 5361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:14:03.908455 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:03.909118 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 5361
I20240503 02:14:03.909329 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:14:03.920384 5427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6: Bootstrap starting.
I20240503 02:14:03.923293 5427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:03.924274 5427 log.cc:824] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:03.926563 5427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6: No bootstrap required, opened a new log
I20240503 02:14:03.932404 5427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385a520856d24cbab1558ef77c93b8c6" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 43347 } }
I20240503 02:14:03.932727 5427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:03.932835 5427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 385a520856d24cbab1558ef77c93b8c6, State: Initialized, Role: FOLLOWER
I20240503 02:14:03.933465 5427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385a520856d24cbab1558ef77c93b8c6" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 43347 } }
I20240503 02:14:03.933697 5427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:03.933836 5427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:03.933982 5427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:03.935953 5427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385a520856d24cbab1558ef77c93b8c6" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 43347 } }
I20240503 02:14:03.936394 5427 leader_election.cc:302] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 385a520856d24cbab1558ef77c93b8c6; no voters:
I20240503 02:14:03.936941 5427 leader_election.cc:288] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:03.937100 5432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:03.938272 5432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [term 1 LEADER]: Becoming Leader. State: Replica: 385a520856d24cbab1558ef77c93b8c6, State: Running, Role: LEADER
I20240503 02:14:03.938724 5432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385a520856d24cbab1558ef77c93b8c6" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 43347 } }
I20240503 02:14:03.938871 5427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:14:03.941243 5433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "385a520856d24cbab1558ef77c93b8c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385a520856d24cbab1558ef77c93b8c6" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 43347 } } }
I20240503 02:14:03.941546 5433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [sys.catalog]: This master's current role is: LEADER
I20240503 02:14:03.941318 5434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 385a520856d24cbab1558ef77c93b8c6. Latest consensus state: current_term: 1 leader_uuid: "385a520856d24cbab1558ef77c93b8c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385a520856d24cbab1558ef77c93b8c6" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 43347 } } }
I20240503 02:14:03.941704 5434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6 [sys.catalog]: This master's current role is: LEADER
I20240503 02:14:03.944577 5438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:14:03.949506 5438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:14:03.957069 5438 catalog_manager.cc:1347] Generated new cluster ID: 6bb2632f9c014e57a1ad54d49a79c5af
I20240503 02:14:03.957170 5438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:14:03.979198 5438 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:14:03.980896 5438 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:14:03.996677 5438 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 385a520856d24cbab1558ef77c93b8c6: Generated new TSK 0
I20240503 02:14:03.997607 5438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:14:04.015100 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.193:0
--local_ip_for_outbound_sockets=127.23.104.193
--webserver_interface=127.23.104.193
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43347
--builtin_ntp_servers=127.23.104.212:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:14:04.337814 5451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:04.338073 5451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:04.338164 5451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:04.345700 5451 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:04.345845 5451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:04.346026 5451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.193
I20240503 02:14:04.354713 5451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.23.104.193
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43347
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.193
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:04.356293 5451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:04.357983 5451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:04.369668 5456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:04.370190 5457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:04.370743 5459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:04.371341 5451 server_base.cc:1034] running on GCE node
I20240503 02:14:04.371887 5451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:04.372764 5451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:04.373960 5451 hybrid_clock.cc:648] HybridClock initialized: now 1714702444373914 us; error 37 us; skew 500 ppm
I20240503 02:14:04.374269 5451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:04.376367 5451 webserver.cc:460] Webserver started at http://127.23.104.193:46705/ using document root <none> and password file <none>
I20240503 02:14:04.377040 5451 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:04.377173 5451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:04.377458 5451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:04.379884 5451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e397c4c6d41d466f977e826a19150699"
format_stamp: "Formatted at 2024-05-03 02:14:04 on dist-test-slave-38ht"
I20240503 02:14:04.380617 5451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e397c4c6d41d466f977e826a19150699"
format_stamp: "Formatted at 2024-05-03 02:14:04 on dist-test-slave-38ht"
I20240503 02:14:04.385182 5451 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:14:04.388123 5465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:04.389302 5451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:14:04.389472 5451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e397c4c6d41d466f977e826a19150699"
format_stamp: "Formatted at 2024-05-03 02:14:04 on dist-test-slave-38ht"
I20240503 02:14:04.389685 5451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:04.411613 5451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:04.412371 5451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:04.412596 5451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:04.413380 5451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:04.414646 5451 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:04.414736 5451 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:04.414811 5451 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:04.414863 5451 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:04.444773 5451 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.193:46169
I20240503 02:14:04.444876 5577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.193:46169 every 8 connection(s)
I20240503 02:14:04.446065 5451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:14:04.453114 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 5451
I20240503 02:14:04.453354 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:14:04.457113 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.194:0
--local_ip_for_outbound_sockets=127.23.104.194
--webserver_interface=127.23.104.194
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43347
--builtin_ntp_servers=127.23.104.212:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:04.458614 5578 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43347
I20240503 02:14:04.458945 5578 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:04.459956 5578 heartbeater.cc:511] Master 127.23.104.254:43347 requested a full tablet report, sending...
I20240503 02:14:04.462132 5392 ts_manager.cc:194] Registered new tserver with Master: e397c4c6d41d466f977e826a19150699 (127.23.104.193:46169)
I20240503 02:14:04.464180 5392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.193:56621
W20240503 02:14:04.781337 5582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:04.781661 5582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:04.781805 5582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:04.789175 5582 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:04.789320 5582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:04.789475 5582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.194
I20240503 02:14:04.798162 5582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.23.104.194
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43347
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.194
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:04.799468 5582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:04.801208 5582 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:04.813235 5587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:04.813654 5588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:04.814169 5590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:04.814819 5582 server_base.cc:1034] running on GCE node
I20240503 02:14:04.815375 5582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:04.816301 5582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:04.817517 5582 hybrid_clock.cc:648] HybridClock initialized: now 1714702444817462 us; error 45 us; skew 500 ppm
I20240503 02:14:04.817823 5582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:04.819947 5582 webserver.cc:460] Webserver started at http://127.23.104.194:39459/ using document root <none> and password file <none>
I20240503 02:14:04.820650 5582 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:04.820770 5582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:04.821059 5582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:04.823468 5582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4bd253f0e6834fda9abbc8ed3cf798a8"
format_stamp: "Formatted at 2024-05-03 02:14:04 on dist-test-slave-38ht"
I20240503 02:14:04.824193 5582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4bd253f0e6834fda9abbc8ed3cf798a8"
format_stamp: "Formatted at 2024-05-03 02:14:04 on dist-test-slave-38ht"
I20240503 02:14:04.829126 5582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240503 02:14:04.832646 5596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:04.833927 5582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 02:14:04.834091 5582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4bd253f0e6834fda9abbc8ed3cf798a8"
format_stamp: "Formatted at 2024-05-03 02:14:04 on dist-test-slave-38ht"
I20240503 02:14:04.834272 5582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:04.855425 5582 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:04.856220 5582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:04.856441 5582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:04.857275 5582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:04.858664 5582 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:04.858744 5582 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:04.858821 5582 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:04.858878 5582 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:04.885972 5582 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.194:34803
I20240503 02:14:04.886047 5708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.194:34803 every 8 connection(s)
I20240503 02:14:04.887156 5582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:14:04.888370 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 5582
I20240503 02:14:04.888566 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:14:04.891830 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.195:0
--local_ip_for_outbound_sockets=127.23.104.195
--webserver_interface=127.23.104.195
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43347
--builtin_ntp_servers=127.23.104.212:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:04.899451 5709 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43347
I20240503 02:14:04.899717 5709 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:04.900452 5709 heartbeater.cc:511] Master 127.23.104.254:43347 requested a full tablet report, sending...
I20240503 02:14:04.901690 5392 ts_manager.cc:194] Registered new tserver with Master: 4bd253f0e6834fda9abbc8ed3cf798a8 (127.23.104.194:34803)
I20240503 02:14:04.902351 5392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.194:51629
W20240503 02:14:05.213980 5712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:05.214246 5712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:05.214331 5712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:05.221441 5712 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:05.221554 5712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:05.221716 5712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.195
I20240503 02:14:05.229945 5712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.23.104.195
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43347
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.195
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:05.231133 5712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:05.232750 5712 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:05.244130 5717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:05.244745 5718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:05.245189 5720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:05.245779 5712 server_base.cc:1034] running on GCE node
I20240503 02:14:05.246321 5712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:05.247155 5712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:05.248340 5712 hybrid_clock.cc:648] HybridClock initialized: now 1714702445248300 us; error 30 us; skew 500 ppm
I20240503 02:14:05.248641 5712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:05.250672 5712 webserver.cc:460] Webserver started at http://127.23.104.195:37575/ using document root <none> and password file <none>
I20240503 02:14:05.251331 5712 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:05.251444 5712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:05.251749 5712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:05.254204 5712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "def13ecc1e2146a2881fe913fcc58e8b"
format_stamp: "Formatted at 2024-05-03 02:14:05 on dist-test-slave-38ht"
I20240503 02:14:05.254925 5712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "def13ecc1e2146a2881fe913fcc58e8b"
format_stamp: "Formatted at 2024-05-03 02:14:05 on dist-test-slave-38ht"
I20240503 02:14:05.259665 5712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20240503 02:14:05.262766 5726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:05.263998 5712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:14:05.264130 5712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "def13ecc1e2146a2881fe913fcc58e8b"
format_stamp: "Formatted at 2024-05-03 02:14:05 on dist-test-slave-38ht"
I20240503 02:14:05.264290 5712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:05.280686 5712 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:05.281453 5712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:05.281690 5712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:05.282503 5712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:05.283887 5712 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:05.283972 5712 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:05.284050 5712 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:05.284104 5712 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:05.312323 5712 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.195:34659
I20240503 02:14:05.312376 5838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.195:34659 every 8 connection(s)
I20240503 02:14:05.313511 5712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:14:05.320109 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 5712
I20240503 02:14:05.320290 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:14:05.323364 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.196:0
--local_ip_for_outbound_sockets=127.23.104.196
--webserver_interface=127.23.104.196
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43347
--builtin_ntp_servers=127.23.104.212:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:05.325238 5839 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43347
I20240503 02:14:05.325534 5839 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:05.326452 5839 heartbeater.cc:511] Master 127.23.104.254:43347 requested a full tablet report, sending...
I20240503 02:14:05.327864 5392 ts_manager.cc:194] Registered new tserver with Master: def13ecc1e2146a2881fe913fcc58e8b (127.23.104.195:34659)
I20240503 02:14:05.328533 5392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.195:39889
I20240503 02:14:05.467379 5578 heartbeater.cc:503] Master 127.23.104.254:43347 was elected leader, sending a full tablet report...
W20240503 02:14:05.642191 5842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:05.642450 5842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:05.642531 5842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:05.650005 5842 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:05.650123 5842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:05.650238 5842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.196
I20240503 02:14:05.658820 5842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.23.104.196
--webserver_port=0
--tserver_master_addrs=127.23.104.254:43347
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.196
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:05.660109 5842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:05.661782 5842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:05.673951 5847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:05.674522 5850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:05.674549 5848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:05.675108 5842 server_base.cc:1034] running on GCE node
I20240503 02:14:05.675716 5842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:05.676681 5842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:05.677899 5842 hybrid_clock.cc:648] HybridClock initialized: now 1714702445677852 us; error 45 us; skew 500 ppm
I20240503 02:14:05.678218 5842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:05.680400 5842 webserver.cc:460] Webserver started at http://127.23.104.196:39375/ using document root <none> and password file <none>
I20240503 02:14:05.681126 5842 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:05.681252 5842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:05.681540 5842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:05.684161 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ea9b6b1eef7c4c598270ec70d6054d90"
format_stamp: "Formatted at 2024-05-03 02:14:05 on dist-test-slave-38ht"
I20240503 02:14:05.684887 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ea9b6b1eef7c4c598270ec70d6054d90"
format_stamp: "Formatted at 2024-05-03 02:14:05 on dist-test-slave-38ht"
I20240503 02:14:05.689632 5842 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:14:05.692804 5856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:05.694072 5842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 02:14:05.694223 5842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ea9b6b1eef7c4c598270ec70d6054d90"
format_stamp: "Formatted at 2024-05-03 02:14:05 on dist-test-slave-38ht"
I20240503 02:14:05.694407 5842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:05.709334 5842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:05.710067 5842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:05.710321 5842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:05.711141 5842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:05.712558 5842 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:05.712653 5842 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:05.712759 5842 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:05.712832 5842 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:05.747292 5842 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.196:39487
I20240503 02:14:05.747360 5969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.196:39487 every 8 connection(s)
I20240503 02:14:05.748581 5842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:14:05.752331 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 5842
I20240503 02:14:05.752513 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:14:05.760318 5970 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43347
I20240503 02:14:05.760545 5970 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:05.761237 5970 heartbeater.cc:511] Master 127.23.104.254:43347 requested a full tablet report, sending...
I20240503 02:14:05.762405 5392 ts_manager.cc:194] Registered new tserver with Master: ea9b6b1eef7c4c598270ec70d6054d90 (127.23.104.196:39487)
I20240503 02:14:05.763095 5392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.196:43347
I20240503 02:14:05.767585 23971 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 02:14:05.782063 5392 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52154:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 02:14:05.807130 5513 tablet_service.cc:1467] Processing CreateTablet for tablet d73fa773896a48a299da2f521f66d0df (DEFAULT_TABLE table=TestTable [id=54dff62f5ce64c60976788efcccb6d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:05.808362 5513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73fa773896a48a299da2f521f66d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:05.817837 5984 tablet_bootstrap.cc:492] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699: Bootstrap starting.
I20240503 02:14:05.820616 5984 tablet_bootstrap.cc:654] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:05.821468 5984 log.cc:824] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:05.823474 5984 tablet_bootstrap.cc:492] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699: No bootstrap required, opened a new log
I20240503 02:14:05.823657 5984 ts_tablet_manager.cc:1367] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:14:05.828985 5984 raft_consensus.cc:348] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:05.829274 5984 raft_consensus.cc:374] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:05.829345 5984 raft_consensus.cc:729] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e397c4c6d41d466f977e826a19150699, State: Initialized, Role: FOLLOWER
I20240503 02:14:05.829823 5984 consensus_queue.cc:260] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:05.829991 5984 raft_consensus.cc:388] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:05.830078 5984 raft_consensus.cc:482] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:05.830179 5984 raft_consensus.cc:3037] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:05.832093 5984 raft_consensus.cc:504] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:05.832587 5984 leader_election.cc:302] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e397c4c6d41d466f977e826a19150699; no voters:
I20240503 02:14:05.833344 5984 leader_election.cc:288] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:05.833500 5986 raft_consensus.cc:2781] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:05.834120 5986 raft_consensus.cc:686] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699 [term 1 LEADER]: Becoming Leader. State: Replica: e397c4c6d41d466f977e826a19150699, State: Running, Role: LEADER
I20240503 02:14:05.834710 5986 consensus_queue.cc:237] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:05.836144 5984 ts_tablet_manager.cc:1398] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20240503 02:14:05.839725 5392 catalog_manager.cc:5484] T d73fa773896a48a299da2f521f66d0df P e397c4c6d41d466f977e826a19150699 reported cstate change: term changed from 0 to 1, leader changed from <none> to e397c4c6d41d466f977e826a19150699 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "e397c4c6d41d466f977e826a19150699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:05.850615 23971 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 02:14:05.852367 23971 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e397c4c6d41d466f977e826a19150699 to finish bootstrapping
I20240503 02:14:05.860112 23971 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4bd253f0e6834fda9abbc8ed3cf798a8 to finish bootstrapping
I20240503 02:14:05.868083 23971 ts_itest-base.cc:233] Waiting for 0 tablets on tserver def13ecc1e2146a2881fe913fcc58e8b to finish bootstrapping
I20240503 02:14:05.875677 23971 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ea9b6b1eef7c4c598270ec70d6054d90 to finish bootstrapping
I20240503 02:14:05.883286 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 5712
I20240503 02:14:05.884037 5834 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:05.904947 5709 heartbeater.cc:503] Master 127.23.104.254:43347 was elected leader, sending a full tablet report...
I20240503 02:14:05.966997 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 5712
I20240503 02:14:05.983006 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 5842
I20240503 02:14:05.984073 5965 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:06.060525 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 5842
I20240503 02:14:09.828832 5392 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52154:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:14:09.839259 5513 tablet_service.cc:1467] Processing CreateTablet for tablet 141b0e0ef2a44e869e99ee7abddcc6a7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5615b632f00e41c2a8e6fadbf3b372e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:09.839710 5513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141b0e0ef2a44e869e99ee7abddcc6a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:09.846161 6004 tablet_bootstrap.cc:492] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699: Bootstrap starting.
I20240503 02:14:09.847973 6004 tablet_bootstrap.cc:654] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:09.850543 6004 tablet_bootstrap.cc:492] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699: No bootstrap required, opened a new log
I20240503 02:14:09.849970 5644 tablet_service.cc:1467] Processing CreateTablet for tablet 2f35d12ac87344919262fc074bf755cd (DEFAULT_TABLE table=rebalance_test_table_0 [id=5615b632f00e41c2a8e6fadbf3b372e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:09.849997 5643 tablet_service.cc:1467] Processing CreateTablet for tablet 702a6b4f013547eb9bf162b5be8b877f (DEFAULT_TABLE table=rebalance_test_table_0 [id=5615b632f00e41c2a8e6fadbf3b372e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:09.850703 6004 ts_tablet_manager.cc:1367] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 02:14:09.851526 5644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f35d12ac87344919262fc074bf755cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:09.851653 6004 raft_consensus.cc:348] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:09.851858 6004 raft_consensus.cc:374] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:09.851960 6004 raft_consensus.cc:729] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e397c4c6d41d466f977e826a19150699, State: Initialized, Role: FOLLOWER
I20240503 02:14:09.852053 5643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702a6b4f013547eb9bf162b5be8b877f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:09.852218 6004 consensus_queue.cc:260] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:09.852393 6004 raft_consensus.cc:388] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:09.852481 6004 raft_consensus.cc:482] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:09.852802 6004 raft_consensus.cc:3037] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:09.855401 6004 raft_consensus.cc:504] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:09.855608 6004 leader_election.cc:302] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e397c4c6d41d466f977e826a19150699; no voters:
I20240503 02:14:09.856053 6004 leader_election.cc:288] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:09.856204 6006 raft_consensus.cc:2781] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:09.856922 6004 ts_tablet_manager.cc:1398] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:14:09.857336 6006 raft_consensus.cc:686] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699 [term 1 LEADER]: Becoming Leader. State: Replica: e397c4c6d41d466f977e826a19150699, State: Running, Role: LEADER
I20240503 02:14:09.857615 6006 consensus_queue.cc:237] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:09.859918 5392 catalog_manager.cc:5484] T 141b0e0ef2a44e869e99ee7abddcc6a7 P e397c4c6d41d466f977e826a19150699 reported cstate change: term changed from 0 to 1, leader changed from <none> to e397c4c6d41d466f977e826a19150699 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "e397c4c6d41d466f977e826a19150699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:09.868844 6011 tablet_bootstrap.cc:492] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8: Bootstrap starting.
I20240503 02:14:09.872061 6011 tablet_bootstrap.cc:654] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:09.873214 6011 log.cc:824] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:09.875669 6011 tablet_bootstrap.cc:492] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8: No bootstrap required, opened a new log
I20240503 02:14:09.875946 6011 ts_tablet_manager.cc:1367] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240503 02:14:09.881719 6011 raft_consensus.cc:348] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:09.882097 6011 raft_consensus.cc:374] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:09.882220 6011 raft_consensus.cc:729] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd253f0e6834fda9abbc8ed3cf798a8, State: Initialized, Role: FOLLOWER
I20240503 02:14:09.882874 6011 consensus_queue.cc:260] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:09.883076 6011 raft_consensus.cc:388] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:09.883203 6011 raft_consensus.cc:482] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:09.883349 6011 raft_consensus.cc:3037] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:09.885285 6011 raft_consensus.cc:504] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:09.885722 6011 leader_election.cc:302] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bd253f0e6834fda9abbc8ed3cf798a8; no voters:
I20240503 02:14:09.886392 6011 leader_election.cc:288] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:09.886508 6013 raft_consensus.cc:2781] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:09.887719 6013 raft_consensus.cc:686] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 LEADER]: Becoming Leader. State: Replica: 4bd253f0e6834fda9abbc8ed3cf798a8, State: Running, Role: LEADER
I20240503 02:14:09.888056 6011 ts_tablet_manager.cc:1398] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20240503 02:14:09.888237 6013 consensus_queue.cc:237] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:09.888484 6011 tablet_bootstrap.cc:492] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8: Bootstrap starting.
I20240503 02:14:09.890286 6011 tablet_bootstrap.cc:654] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8: Neither blocks nor log segments found. Creating new log.
W20240503 02:14:09.892122 5710 tablet.cc:2369] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:14:09.893573 5391 catalog_manager.cc:5484] T 2f35d12ac87344919262fc074bf755cd P 4bd253f0e6834fda9abbc8ed3cf798a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bd253f0e6834fda9abbc8ed3cf798a8 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:09.896585 6011 tablet_bootstrap.cc:492] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8: No bootstrap required, opened a new log
I20240503 02:14:09.896736 6011 ts_tablet_manager.cc:1367] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240503 02:14:09.897737 6011 raft_consensus.cc:348] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:09.897945 6011 raft_consensus.cc:374] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:09.898059 6011 raft_consensus.cc:729] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd253f0e6834fda9abbc8ed3cf798a8, State: Initialized, Role: FOLLOWER
I20240503 02:14:09.898340 6011 consensus_queue.cc:260] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:09.898536 6011 raft_consensus.cc:388] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:09.898617 6011 raft_consensus.cc:482] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:09.898715 6011 raft_consensus.cc:3037] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:09.901588 6011 raft_consensus.cc:504] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:09.901837 6011 leader_election.cc:302] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bd253f0e6834fda9abbc8ed3cf798a8; no voters:
I20240503 02:14:09.902020 6011 leader_election.cc:288] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:09.902107 6013 raft_consensus.cc:2781] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:09.902297 6013 raft_consensus.cc:686] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 LEADER]: Becoming Leader. State: Replica: 4bd253f0e6834fda9abbc8ed3cf798a8, State: Running, Role: LEADER
I20240503 02:14:09.902565 6011 ts_tablet_manager.cc:1398] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20240503 02:14:09.902519 6013 consensus_queue.cc:237] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:09.905989 5391 catalog_manager.cc:5484] T 702a6b4f013547eb9bf162b5be8b877f P 4bd253f0e6834fda9abbc8ed3cf798a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bd253f0e6834fda9abbc8ed3cf798a8 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:11.338961 5391 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52154:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:14:11.346982 5513 tablet_service.cc:1467] Processing CreateTablet for tablet 4225940eb068458a8b5fa88472798fc8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=35db28af2d774e2c8793e63db893d68a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:11.347112 5643 tablet_service.cc:1467] Processing CreateTablet for tablet 749f3a3849fe415d8cb142a4e67ea2c8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=35db28af2d774e2c8793e63db893d68a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:11.347460 5513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4225940eb068458a8b5fa88472798fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:11.347572 5643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 749f3a3849fe415d8cb142a4e67ea2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:11.351958 6045 tablet_bootstrap.cc:492] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699: Bootstrap starting.
I20240503 02:14:11.351962 6046 tablet_bootstrap.cc:492] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8: Bootstrap starting.
I20240503 02:14:11.352082 5512 tablet_service.cc:1467] Processing CreateTablet for tablet 2c89754e42d34504a91853499c882a94 (DEFAULT_TABLE table=rebalance_test_table_1 [id=35db28af2d774e2c8793e63db893d68a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:11.352422 5512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c89754e42d34504a91853499c882a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:11.353446 6046 tablet_bootstrap.cc:654] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:11.353574 6045 tablet_bootstrap.cc:654] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:11.354893 6046 tablet_bootstrap.cc:492] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8: No bootstrap required, opened a new log
I20240503 02:14:11.355013 6046 ts_tablet_manager.cc:1367] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:14:11.355590 6046 raft_consensus.cc:348] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:11.355724 6046 raft_consensus.cc:374] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:11.355840 6046 raft_consensus.cc:729] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd253f0e6834fda9abbc8ed3cf798a8, State: Initialized, Role: FOLLOWER
I20240503 02:14:11.356030 6046 consensus_queue.cc:260] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:11.356181 6046 raft_consensus.cc:388] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:11.356261 6046 raft_consensus.cc:482] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:11.356359 6046 raft_consensus.cc:3037] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:11.358237 6046 raft_consensus.cc:504] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:11.358444 6046 leader_election.cc:302] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bd253f0e6834fda9abbc8ed3cf798a8; no voters:
I20240503 02:14:11.358873 6046 leader_election.cc:288] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:11.359040 6049 raft_consensus.cc:2781] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:11.360608 6046 ts_tablet_manager.cc:1398] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240503 02:14:11.361229 6049 raft_consensus.cc:686] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 LEADER]: Becoming Leader. State: Replica: 4bd253f0e6834fda9abbc8ed3cf798a8, State: Running, Role: LEADER
I20240503 02:14:11.361454 6049 consensus_queue.cc:237] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:11.361697 6045 tablet_bootstrap.cc:492] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699: No bootstrap required, opened a new log
I20240503 02:14:11.361826 6045 ts_tablet_manager.cc:1367] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240503 02:14:11.362617 6045 raft_consensus.cc:348] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:11.362788 6045 raft_consensus.cc:374] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:11.362883 6045 raft_consensus.cc:729] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e397c4c6d41d466f977e826a19150699, State: Initialized, Role: FOLLOWER
I20240503 02:14:11.363057 6045 consensus_queue.cc:260] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:11.363231 6045 raft_consensus.cc:388] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:11.363313 6045 raft_consensus.cc:482] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:11.363420 6045 raft_consensus.cc:3037] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:11.363639 5391 catalog_manager.cc:5484] T 749f3a3849fe415d8cb142a4e67ea2c8 P 4bd253f0e6834fda9abbc8ed3cf798a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bd253f0e6834fda9abbc8ed3cf798a8 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:11.366240 6045 raft_consensus.cc:504] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:11.366488 6045 leader_election.cc:302] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e397c4c6d41d466f977e826a19150699; no voters:
I20240503 02:14:11.366902 6045 leader_election.cc:288] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:11.367066 6054 raft_consensus.cc:2781] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:11.367631 6045 ts_tablet_manager.cc:1398] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240503 02:14:11.367883 6045 tablet_bootstrap.cc:492] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699: Bootstrap starting.
I20240503 02:14:11.369670 6045 tablet_bootstrap.cc:654] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:11.369804 6054 raft_consensus.cc:686] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699 [term 1 LEADER]: Becoming Leader. State: Replica: e397c4c6d41d466f977e826a19150699, State: Running, Role: LEADER
I20240503 02:14:11.370116 6054 consensus_queue.cc:237] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:11.377043 6045 tablet_bootstrap.cc:492] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699: No bootstrap required, opened a new log
I20240503 02:14:11.377190 6045 ts_tablet_manager.cc:1367] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.003s
I20240503 02:14:11.377939 6045 raft_consensus.cc:348] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:11.378183 6045 raft_consensus.cc:374] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:11.378283 6045 raft_consensus.cc:729] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e397c4c6d41d466f977e826a19150699, State: Initialized, Role: FOLLOWER
I20240503 02:14:11.378520 6045 consensus_queue.cc:260] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:11.378710 6045 raft_consensus.cc:388] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:11.378831 6045 raft_consensus.cc:482] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:11.378935 6045 raft_consensus.cc:3037] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:11.380105 5391 catalog_manager.cc:5484] T 4225940eb068458a8b5fa88472798fc8 P e397c4c6d41d466f977e826a19150699 reported cstate change: term changed from 0 to 1, leader changed from <none> to e397c4c6d41d466f977e826a19150699 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "e397c4c6d41d466f977e826a19150699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:11.381356 6045 raft_consensus.cc:504] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:11.381567 6045 leader_election.cc:302] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e397c4c6d41d466f977e826a19150699; no voters:
I20240503 02:14:11.381790 6045 leader_election.cc:288] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:11.381844 6056 raft_consensus.cc:2781] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:11.382000 6056 raft_consensus.cc:686] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699 [term 1 LEADER]: Becoming Leader. State: Replica: e397c4c6d41d466f977e826a19150699, State: Running, Role: LEADER
I20240503 02:14:11.382303 6045 ts_tablet_manager.cc:1398] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20240503 02:14:11.382272 6056 consensus_queue.cc:237] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:11.384639 5391 catalog_manager.cc:5484] T 2c89754e42d34504a91853499c882a94 P e397c4c6d41d466f977e826a19150699 reported cstate change: term changed from 0 to 1, leader changed from <none> to e397c4c6d41d466f977e826a19150699 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "e397c4c6d41d466f977e826a19150699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:11.870817 5391 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52154:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:14:11.877979 5644 tablet_service.cc:1467] Processing CreateTablet for tablet cabfa619fbb740878df0c8c0b7d258b3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=cfd5aab8a23a41b7a4555bfaf0796eb2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:11.877979 5643 tablet_service.cc:1467] Processing CreateTablet for tablet 566459fe8b3a469994e73c0f6e46a183 (DEFAULT_TABLE table=rebalance_test_table_2 [id=cfd5aab8a23a41b7a4555bfaf0796eb2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:11.878465 5644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cabfa619fbb740878df0c8c0b7d258b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:11.878676 5643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566459fe8b3a469994e73c0f6e46a183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:11.879133 5513 tablet_service.cc:1467] Processing CreateTablet for tablet 34ca3a59df62468facc1ae5ffa968ab7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=cfd5aab8a23a41b7a4555bfaf0796eb2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:11.879545 5513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ca3a59df62468facc1ae5ffa968ab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:11.883736 6082 tablet_bootstrap.cc:492] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8: Bootstrap starting.
I20240503 02:14:11.885144 6082 tablet_bootstrap.cc:654] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:11.885746 6083 tablet_bootstrap.cc:492] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699: Bootstrap starting.
I20240503 02:14:11.887048 6082 tablet_bootstrap.cc:492] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8: No bootstrap required, opened a new log
I20240503 02:14:11.887178 6083 tablet_bootstrap.cc:654] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:11.887184 6082 ts_tablet_manager.cc:1367] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:14:11.888195 6082 raft_consensus.cc:348] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:11.888378 6082 raft_consensus.cc:374] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:11.888453 6082 raft_consensus.cc:729] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd253f0e6834fda9abbc8ed3cf798a8, State: Initialized, Role: FOLLOWER
I20240503 02:14:11.888691 6082 consensus_queue.cc:260] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:11.889058 6082 raft_consensus.cc:388] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:11.889153 6082 raft_consensus.cc:482] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:11.889266 6082 raft_consensus.cc:3037] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:11.892000 6082 raft_consensus.cc:504] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:11.892241 6082 leader_election.cc:302] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bd253f0e6834fda9abbc8ed3cf798a8; no voters:
I20240503 02:14:11.893162 6082 leader_election.cc:288] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:11.893514 6086 raft_consensus.cc:2781] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:11.900127 6082 ts_tablet_manager.cc:1398] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20240503 02:14:11.900859 6082 tablet_bootstrap.cc:492] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8: Bootstrap starting.
I20240503 02:14:11.902688 6082 tablet_bootstrap.cc:654] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:11.903163 6086 raft_consensus.cc:686] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 LEADER]: Becoming Leader. State: Replica: 4bd253f0e6834fda9abbc8ed3cf798a8, State: Running, Role: LEADER
I20240503 02:14:11.903681 6086 consensus_queue.cc:237] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:11.906899 6082 tablet_bootstrap.cc:492] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8: No bootstrap required, opened a new log
I20240503 02:14:11.907058 6082 ts_tablet_manager.cc:1367] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240503 02:14:11.907811 6082 raft_consensus.cc:348] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:11.908002 6082 raft_consensus.cc:374] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:11.908082 6082 raft_consensus.cc:729] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd253f0e6834fda9abbc8ed3cf798a8, State: Initialized, Role: FOLLOWER
I20240503 02:14:11.908329 6082 consensus_queue.cc:260] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:11.908504 6082 raft_consensus.cc:388] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:11.908600 6082 raft_consensus.cc:482] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:11.908689 6082 raft_consensus.cc:3037] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:11.910527 6082 raft_consensus.cc:504] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:11.910838 6082 leader_election.cc:302] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bd253f0e6834fda9abbc8ed3cf798a8; no voters:
I20240503 02:14:11.911067 6082 leader_election.cc:288] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:11.911227 6088 raft_consensus.cc:2781] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:11.911589 6082 ts_tablet_manager.cc:1398] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20240503 02:14:11.914088 5391 catalog_manager.cc:5484] T 566459fe8b3a469994e73c0f6e46a183 P 4bd253f0e6834fda9abbc8ed3cf798a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bd253f0e6834fda9abbc8ed3cf798a8 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:11.920126 6088 raft_consensus.cc:686] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 LEADER]: Becoming Leader. State: Replica: 4bd253f0e6834fda9abbc8ed3cf798a8, State: Running, Role: LEADER
I20240503 02:14:11.920466 6088 consensus_queue.cc:237] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:11.921015 6083 tablet_bootstrap.cc:492] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699: No bootstrap required, opened a new log
I20240503 02:14:11.921212 6083 ts_tablet_manager.cc:1367] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.013s
I20240503 02:14:11.922261 6083 raft_consensus.cc:348] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:11.922502 6083 raft_consensus.cc:374] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:11.922602 6083 raft_consensus.cc:729] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e397c4c6d41d466f977e826a19150699, State: Initialized, Role: FOLLOWER
I20240503 02:14:11.922791 5391 catalog_manager.cc:5484] T cabfa619fbb740878df0c8c0b7d258b3 P 4bd253f0e6834fda9abbc8ed3cf798a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bd253f0e6834fda9abbc8ed3cf798a8 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:11.922878 6083 consensus_queue.cc:260] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:11.923113 6083 raft_consensus.cc:388] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:11.923197 6083 raft_consensus.cc:482] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:11.923311 6083 raft_consensus.cc:3037] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:11.927043 6083 raft_consensus.cc:504] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:11.927392 6083 leader_election.cc:302] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e397c4c6d41d466f977e826a19150699; no voters:
I20240503 02:14:11.927641 6083 leader_election.cc:288] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:11.927757 6056 raft_consensus.cc:2781] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:11.931753 6056 raft_consensus.cc:686] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699 [term 1 LEADER]: Becoming Leader. State: Replica: e397c4c6d41d466f977e826a19150699, State: Running, Role: LEADER
I20240503 02:14:11.932220 6056 consensus_queue.cc:237] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:11.932504 6083 ts_tablet_manager.cc:1398] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20240503 02:14:11.934644 5391 catalog_manager.cc:5484] T 34ca3a59df62468facc1ae5ffa968ab7 P e397c4c6d41d466f977e826a19150699 reported cstate change: term changed from 0 to 1, leader changed from <none> to e397c4c6d41d466f977e826a19150699 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "e397c4c6d41d466f977e826a19150699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:12.424568 5391 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52154:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:14:12.431905 5513 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4373cd5a3345e7aa3d88d41796e3ab (DEFAULT_TABLE table=rebalance_test_table_3 [id=abcb1f2039ff492688732af1647da670]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:12.431998 5644 tablet_service.cc:1467] Processing CreateTablet for tablet bf2a429cb2324082846b265ea43b8ab8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=abcb1f2039ff492688732af1647da670]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:12.432183 5512 tablet_service.cc:1467] Processing CreateTablet for tablet 162b96bc72e54f6fb28670d49a7896de (DEFAULT_TABLE table=rebalance_test_table_3 [id=abcb1f2039ff492688732af1647da670]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:12.432411 5644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf2a429cb2324082846b265ea43b8ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:12.432408 5513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4373cd5a3345e7aa3d88d41796e3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:12.432581 5512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162b96bc72e54f6fb28670d49a7896de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:12.437446 6120 tablet_bootstrap.cc:492] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699: Bootstrap starting.
I20240503 02:14:12.438733 6121 tablet_bootstrap.cc:492] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8: Bootstrap starting.
I20240503 02:14:12.439235 6120 tablet_bootstrap.cc:654] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:12.440713 6121 tablet_bootstrap.cc:654] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:12.441510 6120 tablet_bootstrap.cc:492] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699: No bootstrap required, opened a new log
I20240503 02:14:12.441641 6120 ts_tablet_manager.cc:1367] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:14:12.442176 6120 raft_consensus.cc:348] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:12.442313 6120 raft_consensus.cc:374] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:12.442320 6121 tablet_bootstrap.cc:492] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8: No bootstrap required, opened a new log
I20240503 02:14:12.442409 6120 raft_consensus.cc:729] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e397c4c6d41d466f977e826a19150699, State: Initialized, Role: FOLLOWER
I20240503 02:14:12.442442 6121 ts_tablet_manager.cc:1367] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:14:12.442618 6120 consensus_queue.cc:260] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:12.442798 6120 raft_consensus.cc:388] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:12.442873 6120 raft_consensus.cc:482] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:12.442972 6120 raft_consensus.cc:3037] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:12.443042 6121 raft_consensus.cc:348] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:12.443194 6121 raft_consensus.cc:374] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:12.443289 6121 raft_consensus.cc:729] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd253f0e6834fda9abbc8ed3cf798a8, State: Initialized, Role: FOLLOWER
I20240503 02:14:12.443498 6121 consensus_queue.cc:260] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:12.443672 6121 raft_consensus.cc:388] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:12.443750 6121 raft_consensus.cc:482] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:12.443902 6121 raft_consensus.cc:3037] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:12.445219 6120 raft_consensus.cc:504] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:12.445469 6120 leader_election.cc:302] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e397c4c6d41d466f977e826a19150699; no voters:
I20240503 02:14:12.445950 6120 leader_election.cc:288] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:12.446151 6124 raft_consensus.cc:2781] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:12.446314 6121 raft_consensus.cc:504] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:12.446557 6121 leader_election.cc:302] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bd253f0e6834fda9abbc8ed3cf798a8; no voters:
I20240503 02:14:12.446780 6088 raft_consensus.cc:2781] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:12.452775 6121 leader_election.cc:288] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:12.452826 6124 raft_consensus.cc:686] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699 [term 1 LEADER]: Becoming Leader. State: Replica: e397c4c6d41d466f977e826a19150699, State: Running, Role: LEADER
I20240503 02:14:12.453732 6124 consensus_queue.cc:237] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:12.465435 5392 catalog_manager.cc:5484] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699 reported cstate change: term changed from 0 to 1, leader changed from <none> to e397c4c6d41d466f977e826a19150699 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "e397c4c6d41d466f977e826a19150699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:12.465873 6121 ts_tablet_manager.cc:1398] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8: Time spent starting tablet: real 0.023s user 0.004s sys 0.007s
I20240503 02:14:12.466064 6088 raft_consensus.cc:686] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [term 1 LEADER]: Becoming Leader. State: Replica: 4bd253f0e6834fda9abbc8ed3cf798a8, State: Running, Role: LEADER
I20240503 02:14:12.466298 6088 consensus_queue.cc:237] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } }
I20240503 02:14:12.472537 6120 ts_tablet_manager.cc:1398] T 2a4373cd5a3345e7aa3d88d41796e3ab P e397c4c6d41d466f977e826a19150699: Time spent starting tablet: real 0.031s user 0.011s sys 0.019s
I20240503 02:14:12.472847 6120 tablet_bootstrap.cc:492] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699: Bootstrap starting.
I20240503 02:14:12.473155 5392 catalog_manager.cc:5484] T bf2a429cb2324082846b265ea43b8ab8 P 4bd253f0e6834fda9abbc8ed3cf798a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bd253f0e6834fda9abbc8ed3cf798a8 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd253f0e6834fda9abbc8ed3cf798a8" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 34803 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:12.474601 6120 tablet_bootstrap.cc:654] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:12.476678 6120 tablet_bootstrap.cc:492] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699: No bootstrap required, opened a new log
I20240503 02:14:12.477041 6120 ts_tablet_manager.cc:1367] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 02:14:12.478144 6120 raft_consensus.cc:348] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:12.478346 6120 raft_consensus.cc:374] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:12.478418 6120 raft_consensus.cc:729] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e397c4c6d41d466f977e826a19150699, State: Initialized, Role: FOLLOWER
I20240503 02:14:12.478621 6120 consensus_queue.cc:260] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:12.478775 6120 raft_consensus.cc:388] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:12.478878 6120 raft_consensus.cc:482] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:12.478977 6120 raft_consensus.cc:3037] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:12.481916 6120 raft_consensus.cc:504] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:12.482136 6120 leader_election.cc:302] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e397c4c6d41d466f977e826a19150699; no voters:
I20240503 02:14:12.482343 6120 leader_election.cc:288] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:12.482434 6125 raft_consensus.cc:2781] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:12.482671 6125 raft_consensus.cc:686] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699 [term 1 LEADER]: Becoming Leader. State: Replica: e397c4c6d41d466f977e826a19150699, State: Running, Role: LEADER
I20240503 02:14:12.482764 6120 ts_tablet_manager.cc:1398] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 02:14:12.482973 6125 consensus_queue.cc:237] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } }
I20240503 02:14:12.485244 5391 catalog_manager.cc:5484] T 162b96bc72e54f6fb28670d49a7896de P e397c4c6d41d466f977e826a19150699 reported cstate change: term changed from 0 to 1, leader changed from <none> to e397c4c6d41d466f977e826a19150699 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "e397c4c6d41d466f977e826a19150699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397c4c6d41d466f977e826a19150699" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 46169 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:12.980214 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.195:34659
--local_ip_for_outbound_sockets=127.23.104.195
--tserver_master_addrs=127.23.104.254:43347
--webserver_port=37575
--webserver_interface=127.23.104.195
--builtin_ntp_servers=127.23.104.212:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:14:13.312935 6156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:13.313202 6156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:13.313289 6156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:13.320556 6156 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:13.320686 6156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:13.320839 6156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.195
I20240503 02:14:13.330296 6156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.195:34659
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.23.104.195
--webserver_port=37575
--tserver_master_addrs=127.23.104.254:43347
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.195
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:13.331548 6156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:13.333318 6156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:13.345077 6161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:13.345521 6164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:13.346017 6162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:13.346598 6156 server_base.cc:1034] running on GCE node
I20240503 02:14:13.347122 6156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:13.348040 6156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:13.349238 6156 hybrid_clock.cc:648] HybridClock initialized: now 1714702453349196 us; error 39 us; skew 500 ppm
I20240503 02:14:13.349531 6156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:13.351590 6156 webserver.cc:460] Webserver started at http://127.23.104.195:37575/ using document root <none> and password file <none>
I20240503 02:14:13.352312 6156 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:13.352454 6156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:13.357159 6156 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:14:13.359602 6170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:13.360811 6156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:14:13.360967 6156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "def13ecc1e2146a2881fe913fcc58e8b"
format_stamp: "Formatted at 2024-05-03 02:14:05 on dist-test-slave-38ht"
I20240503 02:14:13.361708 6156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:13.378870 6156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:13.379602 6156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:13.379915 6156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:13.380761 6156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:13.382146 6156 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:13.382226 6156 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:13.382350 6156 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:13.382405 6156 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:13.412814 6156 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.195:34659
I20240503 02:14:13.412864 6283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.195:34659 every 8 connection(s)
I20240503 02:14:13.414232 6156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:14:13.424218 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 6156
I20240503 02:14:13.425027 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.196:39487
--local_ip_for_outbound_sockets=127.23.104.196
--tserver_master_addrs=127.23.104.254:43347
--webserver_port=39375
--webserver_interface=127.23.104.196
--builtin_ntp_servers=127.23.104.212:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:13.425951 6284 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43347
I20240503 02:14:13.426218 6284 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:13.427035 6284 heartbeater.cc:511] Master 127.23.104.254:43347 requested a full tablet report, sending...
I20240503 02:14:13.428484 5391 ts_manager.cc:194] Re-registered known tserver with Master: def13ecc1e2146a2881fe913fcc58e8b (127.23.104.195:34659)
I20240503 02:14:13.429283 5391 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.195:47983
W20240503 02:14:13.751730 6288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:13.751998 6288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:13.752144 6288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:13.758903 6288 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:13.759009 6288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:13.759196 6288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.196
I20240503 02:14:13.767473 6288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.196:39487
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.23.104.196
--webserver_port=39375
--tserver_master_addrs=127.23.104.254:43347
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.196
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:13.768677 6288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:13.770395 6288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:13.781667 6293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:13.782217 6294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:13.782663 6296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:13.783346 6288 server_base.cc:1034] running on GCE node
I20240503 02:14:13.783859 6288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:13.784731 6288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:13.785991 6288 hybrid_clock.cc:648] HybridClock initialized: now 1714702453785924 us; error 65 us; skew 500 ppm
I20240503 02:14:13.786273 6288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:13.788270 6288 webserver.cc:460] Webserver started at http://127.23.104.196:39375/ using document root <none> and password file <none>
I20240503 02:14:13.788923 6288 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:13.789049 6288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:13.793720 6288 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:14:13.796180 6302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:13.797320 6288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:14:13.797454 6288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ea9b6b1eef7c4c598270ec70d6054d90"
format_stamp: "Formatted at 2024-05-03 02:14:05 on dist-test-slave-38ht"
I20240503 02:14:13.798169 6288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:13.817451 6288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:13.818181 6288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:13.818442 6288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:13.819278 6288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:13.820658 6288 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:13.820737 6288 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:13.820878 6288 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:13.820931 6288 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:13.851202 6288 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.196:39487
I20240503 02:14:13.851236 6414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.196:39487 every 8 connection(s)
I20240503 02:14:13.852510 6288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:14:13.853703 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 6288
I20240503 02:14:13.868918 6415 heartbeater.cc:347] Connected to a master server at 127.23.104.254:43347
I20240503 02:14:13.869167 6415 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:13.869912 6415 heartbeater.cc:511] Master 127.23.104.254:43347 requested a full tablet report, sending...
I20240503 02:14:13.871305 5391 ts_manager.cc:194] Re-registered known tserver with Master: ea9b6b1eef7c4c598270ec70d6054d90 (127.23.104.196:39487)
I20240503 02:14:13.872088 5391 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.196:35429
I20240503 02:14:14.315773 5512 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:14:14.316829 5644 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:14:14.323935 6218 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:14.334631 6350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:14.431541 6284 heartbeater.cc:503] Master 127.23.104.254:43347 was elected leader, sending a full tablet report...
I20240503 02:14:14.496338 5512 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:14:14.497280 5644 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:14:14.500078 6218 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:14.500990 6350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:14.755196 5644 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:14:14.756935 6218 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:14.764420 6350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:14.767001 5512 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
385a520856d24cbab1558ef77c93b8c6 | 127.23.104.254:43347 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.23.104.212:37283 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
4bd253f0e6834fda9abbc8ed3cf798a8 | 127.23.104.194:34803 | HEALTHY | <none> | 6 | 0
def13ecc1e2146a2881fe913fcc58e8b | 127.23.104.195:34659 | HEALTHY | <none> | 0 | 0
e397c4c6d41d466f977e826a19150699 | 127.23.104.193:46169 | HEALTHY | <none> | 7 | 0
ea9b6b1eef7c4c598270ec70d6054d90 | 127.23.104.196:39487 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.23.104.193 | experimental | 127.23.104.193:46169
local_ip_for_outbound_sockets | 127.23.104.194 | experimental | 127.23.104.194:34803
local_ip_for_outbound_sockets | 127.23.104.195 | experimental | 127.23.104.195:34659
local_ip_for_outbound_sockets | 127.23.104.196 | experimental | 127.23.104.196:39487
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.104.193:46169
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.23.104.194:34803
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.104.195:34659
server_dump_info_path | /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.23.104.196:39487
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.23.104.212:37283 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 6
Third Quartile | 7
Maximum | 7
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 02:14:14.810508 23971 log_verifier.cc:126] Checking tablet 141b0e0ef2a44e869e99ee7abddcc6a7
I20240503 02:14:14.818058 23971 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 141b0e0ef2a44e869e99ee7abddcc6a7
I20240503 02:14:14.818164 23971 log_verifier.cc:126] Checking tablet 162b96bc72e54f6fb28670d49a7896de
I20240503 02:14:14.858429 23971 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 162b96bc72e54f6fb28670d49a7896de
I20240503 02:14:14.858635 23971 log_verifier.cc:126] Checking tablet 2a4373cd5a3345e7aa3d88d41796e3ab
I20240503 02:14:14.867580 23971 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 2a4373cd5a3345e7aa3d88d41796e3ab
I20240503 02:14:14.867671 23971 log_verifier.cc:126] Checking tablet 2c89754e42d34504a91853499c882a94
I20240503 02:14:14.874255 6415 heartbeater.cc:503] Master 127.23.104.254:43347 was elected leader, sending a full tablet report...
I20240503 02:14:14.874961 23971 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 2c89754e42d34504a91853499c882a94
I20240503 02:14:14.875059 23971 log_verifier.cc:126] Checking tablet 2f35d12ac87344919262fc074bf755cd
I20240503 02:14:14.882676 23971 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 2f35d12ac87344919262fc074bf755cd
I20240503 02:14:14.882763 23971 log_verifier.cc:126] Checking tablet 34ca3a59df62468facc1ae5ffa968ab7
I20240503 02:14:14.889654 23971 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 34ca3a59df62468facc1ae5ffa968ab7
I20240503 02:14:14.889744 23971 log_verifier.cc:126] Checking tablet 4225940eb068458a8b5fa88472798fc8
I20240503 02:14:14.897167 23971 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 4225940eb068458a8b5fa88472798fc8
I20240503 02:14:14.897261 23971 log_verifier.cc:126] Checking tablet 566459fe8b3a469994e73c0f6e46a183
I20240503 02:14:14.904978 23971 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 566459fe8b3a469994e73c0f6e46a183
I20240503 02:14:14.905103 23971 log_verifier.cc:126] Checking tablet 702a6b4f013547eb9bf162b5be8b877f
I20240503 02:14:14.914602 23971 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 702a6b4f013547eb9bf162b5be8b877f
I20240503 02:14:14.914710 23971 log_verifier.cc:126] Checking tablet 749f3a3849fe415d8cb142a4e67ea2c8
I20240503 02:14:14.924477 23971 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 749f3a3849fe415d8cb142a4e67ea2c8
I20240503 02:14:14.924582 23971 log_verifier.cc:126] Checking tablet bf2a429cb2324082846b265ea43b8ab8
I20240503 02:14:14.932837 23971 log_verifier.cc:177] Verified matching terms for 1 ops in tablet bf2a429cb2324082846b265ea43b8ab8
I20240503 02:14:14.932933 23971 log_verifier.cc:126] Checking tablet cabfa619fbb740878df0c8c0b7d258b3
I20240503 02:14:14.940100 23971 log_verifier.cc:177] Verified matching terms for 1 ops in tablet cabfa619fbb740878df0c8c0b7d258b3
I20240503 02:14:14.940196 23971 log_verifier.cc:126] Checking tablet d73fa773896a48a299da2f521f66d0df
I20240503 02:14:14.947213 23971 log_verifier.cc:177] Verified matching terms for 1 ops in tablet d73fa773896a48a299da2f521f66d0df
I20240503 02:14:14.948092 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 5451
I20240503 02:14:14.949285 5573 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:15.041954 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 5451
I20240503 02:14:15.060950 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 5582
I20240503 02:14:15.061978 5703 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:15.154745 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 5582
I20240503 02:14:15.173431 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 6156
I20240503 02:14:15.179189 6279 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:15.263109 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 6156
I20240503 02:14:15.280311 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 6288
I20240503 02:14:15.285763 6410 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:15.369748 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 6288
I20240503 02:14:15.386807 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 5361
I20240503 02:14:15.387769 5422 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:15.497262 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 5361
2024-05-03T02:14:15Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5 (12106 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (12106 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1
I20240503 02:14:15.551084 23971 test_util.cc:268] Using random seed: 1187132488
I20240503 02:14:15.552868 23971 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:14:15.552942 23971 ts_itest-base.cc:113] --------------
I20240503 02:14:15.552979 23971 ts_itest-base.cc:114] 4 tablet servers
I20240503 02:14:15.553017 23971 ts_itest-base.cc:115] 2 replicas per TS
I20240503 02:14:15.553052 23971 ts_itest-base.cc:116] --------------
2024-05-03T02:14:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:14:15Z Disabled control of system clock
I20240503 02:14:15.597908 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.104.254:40161
--webserver_interface=127.23.104.254
--webserver_port=0
--builtin_ntp_servers=127.23.104.212:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.104.254:40161
--location_mapping_cmd=/tmp/dist-test-taskTHQSCq/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 02:14:15.910688 6532 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 02:14:15.910970 6532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:15.911046 6532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:15.911145 6532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:15.918080 6532 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:15.918233 6532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:14:15.918359 6532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:15.918428 6532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:14:15.918514 6532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:14:15.926937 6532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskTHQSCq/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--ipki_ca_key_size=768
--master_addresses=127.23.104.254:40161
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.104.254:40161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.23.104.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:15.928056 6532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:15.929582 6532 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:15.939595 6540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:15.940160 6537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:15.940690 6538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:15.941258 6532 server_base.cc:1034] running on GCE node
I20240503 02:14:15.941879 6532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:15.943040 6532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:15.944243 6532 hybrid_clock.cc:648] HybridClock initialized: now 1714702455944192 us; error 27 us; skew 500 ppm
I20240503 02:14:15.944569 6532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:15.951768 6532 webserver.cc:460] Webserver started at http://127.23.104.254:43609/ using document root <none> and password file <none>
I20240503 02:14:15.952494 6532 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:15.952611 6532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:15.952868 6532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:15.955159 6532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6565746845da411e8947ba0439c0ed30"
format_stamp: "Formatted at 2024-05-03 02:14:15 on dist-test-slave-38ht"
I20240503 02:14:15.955875 6532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6565746845da411e8947ba0439c0ed30"
format_stamp: "Formatted at 2024-05-03 02:14:15 on dist-test-slave-38ht"
I20240503 02:14:15.960513 6532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 02:14:15.963464 6546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:15.964610 6532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:14:15.964753 6532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6565746845da411e8947ba0439c0ed30"
format_stamp: "Formatted at 2024-05-03 02:14:15 on dist-test-slave-38ht"
I20240503 02:14:15.964900 6532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:15.983129 6532 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:15.983852 6532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:15.984081 6532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:16.000730 6532 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.254:40161
I20240503 02:14:16.000764 6598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.254:40161 every 8 connection(s)
I20240503 02:14:16.001873 6532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:14:16.003048 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 6532
I20240503 02:14:16.003348 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:14:16.006104 6599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:16.017794 6599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30: Bootstrap starting.
I20240503 02:14:16.020514 6599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:16.021412 6599 log.cc:824] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:16.023694 6599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30: No bootstrap required, opened a new log
I20240503 02:14:16.029126 6599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6565746845da411e8947ba0439c0ed30" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 40161 } }
I20240503 02:14:16.029414 6599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:16.029477 6599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6565746845da411e8947ba0439c0ed30, State: Initialized, Role: FOLLOWER
I20240503 02:14:16.029976 6599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6565746845da411e8947ba0439c0ed30" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 40161 } }
I20240503 02:14:16.030148 6599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:16.030239 6599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:16.030360 6599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:16.032203 6599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6565746845da411e8947ba0439c0ed30" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 40161 } }
I20240503 02:14:16.032613 6599 leader_election.cc:302] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6565746845da411e8947ba0439c0ed30; no voters:
I20240503 02:14:16.033257 6599 leader_election.cc:288] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:16.033347 6604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:16.033999 6604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [term 1 LEADER]: Becoming Leader. State: Replica: 6565746845da411e8947ba0439c0ed30, State: Running, Role: LEADER
I20240503 02:14:16.034502 6604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6565746845da411e8947ba0439c0ed30" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 40161 } }
I20240503 02:14:16.035041 6599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:14:16.037082 6606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6565746845da411e8947ba0439c0ed30. Latest consensus state: current_term: 1 leader_uuid: "6565746845da411e8947ba0439c0ed30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6565746845da411e8947ba0439c0ed30" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 40161 } } }
I20240503 02:14:16.037058 6605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6565746845da411e8947ba0439c0ed30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6565746845da411e8947ba0439c0ed30" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 40161 } } }
I20240503 02:14:16.037389 6606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [sys.catalog]: This master's current role is: LEADER
I20240503 02:14:16.037425 6605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30 [sys.catalog]: This master's current role is: LEADER
W20240503 02:14:16.045204 6620 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:14:16.045289 6620 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:14:16.045368 6609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:14:16.046938 6609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:14:16.053479 6609 catalog_manager.cc:1347] Generated new cluster ID: 3e3f901817564e88b14049c74e930b55
I20240503 02:14:16.053572 6609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:14:16.091682 6609 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:14:16.093176 6609 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:14:16.102880 6609 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6565746845da411e8947ba0439c0ed30: Generated new TSK 0
I20240503 02:14:16.103749 6609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:14:16.111050 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.193:0
--local_ip_for_outbound_sockets=127.23.104.193
--webserver_interface=127.23.104.193
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40161
--builtin_ntp_servers=127.23.104.212:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:14:16.428733 6623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:16.429006 6623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:16.429141 6623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:16.436548 6623 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:16.436669 6623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:16.436843 6623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.193
I20240503 02:14:16.445587 6623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.23.104.193
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.193
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:16.446857 6623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:16.448626 6623 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:16.460529 6631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:16.461097 6629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:16.461586 6628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:16.462251 6623 server_base.cc:1034] running on GCE node
I20240503 02:14:16.462824 6623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:16.463703 6623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:16.464898 6623 hybrid_clock.cc:648] HybridClock initialized: now 1714702456464845 us; error 48 us; skew 500 ppm
I20240503 02:14:16.465183 6623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:16.467367 6623 webserver.cc:460] Webserver started at http://127.23.104.193:41749/ using document root <none> and password file <none>
I20240503 02:14:16.468135 6623 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:16.468252 6623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:16.468562 6623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:16.471124 6623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b1ff4710a0a24719ae6a43233f261fb8"
format_stamp: "Formatted at 2024-05-03 02:14:16 on dist-test-slave-38ht"
I20240503 02:14:16.471895 6623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b1ff4710a0a24719ae6a43233f261fb8"
format_stamp: "Formatted at 2024-05-03 02:14:16 on dist-test-slave-38ht"
I20240503 02:14:16.476950 6623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20240503 02:14:16.480262 6637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:16.481604 6623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240503 02:14:16.481757 6623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b1ff4710a0a24719ae6a43233f261fb8"
format_stamp: "Formatted at 2024-05-03 02:14:16 on dist-test-slave-38ht"
I20240503 02:14:16.481948 6623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:16.511411 6623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:16.512248 6623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:16.512518 6623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:16.513433 6623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:16.514838 6623 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:16.514937 6623 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:16.515055 6623 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:16.515125 6623 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:16.545529 6623 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.193:36019
I20240503 02:14:16.545590 6750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.193:36019 every 8 connection(s)
I20240503 02:14:16.546715 6623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:14:16.552749 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 6623
I20240503 02:14:16.552963 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:14:16.556567 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.194:0
--local_ip_for_outbound_sockets=127.23.104.194
--webserver_interface=127.23.104.194
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40161
--builtin_ntp_servers=127.23.104.212:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:16.559818 6751 heartbeater.cc:347] Connected to a master server at 127.23.104.254:40161
I20240503 02:14:16.560099 6751 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:16.560864 6751 heartbeater.cc:511] Master 127.23.104.254:40161 requested a full tablet report, sending...
I20240503 02:14:16.615339 6563 ts_manager.cc:194] Registered new tserver with Master: b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019)
I20240503 02:14:16.617791 6563 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.193:50851
W20240503 02:14:16.884991 6755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:16.885244 6755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:16.885342 6755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:16.892418 6755 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:16.892521 6755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:16.892629 6755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.194
I20240503 02:14:16.900681 6755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.23.104.194
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.194
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:16.901844 6755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:16.903447 6755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:16.915490 6764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:16.916018 6762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:16.916618 6761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:16.917176 6755 server_base.cc:1034] running on GCE node
I20240503 02:14:16.917749 6755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:16.918620 6755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:16.919915 6755 hybrid_clock.cc:648] HybridClock initialized: now 1714702456919868 us; error 43 us; skew 500 ppm
I20240503 02:14:16.920228 6755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:16.922400 6755 webserver.cc:460] Webserver started at http://127.23.104.194:35565/ using document root <none> and password file <none>
I20240503 02:14:16.923079 6755 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:16.923192 6755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:16.923489 6755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:16.926038 6755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ee60d5902a7044f9b55e6702b4d8a29a"
format_stamp: "Formatted at 2024-05-03 02:14:16 on dist-test-slave-38ht"
I20240503 02:14:16.926824 6755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ee60d5902a7044f9b55e6702b4d8a29a"
format_stamp: "Formatted at 2024-05-03 02:14:16 on dist-test-slave-38ht"
I20240503 02:14:16.931716 6755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20240503 02:14:16.934839 6770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:16.936144 6755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 02:14:16.936306 6755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ee60d5902a7044f9b55e6702b4d8a29a"
format_stamp: "Formatted at 2024-05-03 02:14:16 on dist-test-slave-38ht"
I20240503 02:14:16.936487 6755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:16.961877 6755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:16.962610 6755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:16.962862 6755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:16.963701 6755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:16.965036 6755 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:16.965131 6755 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:16.965248 6755 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:16.965332 6755 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:16.995626 6755 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.194:33215
I20240503 02:14:16.995678 6883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.194:33215 every 8 connection(s)
I20240503 02:14:16.996852 6755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:14:17.005046 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 6755
I20240503 02:14:17.005239 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:14:17.008327 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.195:0
--local_ip_for_outbound_sockets=127.23.104.195
--webserver_interface=127.23.104.195
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40161
--builtin_ntp_servers=127.23.104.212:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:17.009037 6884 heartbeater.cc:347] Connected to a master server at 127.23.104.254:40161
I20240503 02:14:17.009305 6884 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:17.010201 6884 heartbeater.cc:511] Master 127.23.104.254:40161 requested a full tablet report, sending...
I20240503 02:14:17.057518 6563 ts_manager.cc:194] Registered new tserver with Master: ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194:33215)
I20240503 02:14:17.058396 6563 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.194:38753
W20240503 02:14:17.330072 6888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:17.330332 6888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:17.330464 6888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:17.337630 6888 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:17.337749 6888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:17.337888 6888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.195
I20240503 02:14:17.346657 6888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.23.104.195
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.195
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:17.347884 6888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:17.349478 6888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:17.361220 6893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:17.361763 6896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:17.362267 6894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:17.362936 6888 server_base.cc:1034] running on GCE node
I20240503 02:14:17.363417 6888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:17.364238 6888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:17.365422 6888 hybrid_clock.cc:648] HybridClock initialized: now 1714702457365381 us; error 31 us; skew 500 ppm
I20240503 02:14:17.365707 6888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:17.367766 6888 webserver.cc:460] Webserver started at http://127.23.104.195:46287/ using document root <none> and password file <none>
I20240503 02:14:17.368479 6888 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:17.368574 6888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:17.368876 6888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:17.371378 6888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4fbc402737f64c5b9297ff0f7e22044e"
format_stamp: "Formatted at 2024-05-03 02:14:17 on dist-test-slave-38ht"
I20240503 02:14:17.372116 6888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4fbc402737f64c5b9297ff0f7e22044e"
format_stamp: "Formatted at 2024-05-03 02:14:17 on dist-test-slave-38ht"
I20240503 02:14:17.376822 6888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240503 02:14:17.379978 6902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:17.381167 6888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:14:17.381304 6888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4fbc402737f64c5b9297ff0f7e22044e"
format_stamp: "Formatted at 2024-05-03 02:14:17 on dist-test-slave-38ht"
I20240503 02:14:17.381496 6888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:17.400781 6888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:17.401556 6888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:17.401809 6888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:17.402678 6888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:17.404096 6888 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:17.404183 6888 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:17.404299 6888 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:17.404377 6888 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:17.434556 6888 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.195:32971
I20240503 02:14:17.434599 7014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.195:32971 every 8 connection(s)
I20240503 02:14:17.435981 6888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:14:17.439003 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 6888
I20240503 02:14:17.439172 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:14:17.442936 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.196:0
--local_ip_for_outbound_sockets=127.23.104.196
--webserver_interface=127.23.104.196
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40161
--builtin_ntp_servers=127.23.104.212:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:17.449326 7015 heartbeater.cc:347] Connected to a master server at 127.23.104.254:40161
I20240503 02:14:17.449642 7015 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:17.450582 7015 heartbeater.cc:511] Master 127.23.104.254:40161 requested a full tablet report, sending...
I20240503 02:14:17.509531 6563 ts_manager.cc:194] Registered new tserver with Master: 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195:32971)
I20240503 02:14:17.510475 6563 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.195:39905
I20240503 02:14:17.621107 6751 heartbeater.cc:503] Master 127.23.104.254:40161 was elected leader, sending a full tablet report...
W20240503 02:14:17.785282 7018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:17.785529 7018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:17.785645 7018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:17.792891 7018 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:17.792992 7018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:17.793087 7018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.196
I20240503 02:14:17.801474 7018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.23.104.196
--webserver_port=0
--tserver_master_addrs=127.23.104.254:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.196
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:17.802548 7018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:17.804163 7018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:17.815563 7025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:17.815773 7024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:17.816323 7027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:17.816928 7018 server_base.cc:1034] running on GCE node
I20240503 02:14:17.817473 7018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:17.818289 7018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:17.819602 7018 hybrid_clock.cc:648] HybridClock initialized: now 1714702457819557 us; error 39 us; skew 500 ppm
I20240503 02:14:17.819968 7018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:17.822057 7018 webserver.cc:460] Webserver started at http://127.23.104.196:42609/ using document root <none> and password file <none>
I20240503 02:14:17.822718 7018 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:17.822837 7018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:17.823161 7018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:17.825618 7018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4334b0381be846cd801049e113b85b75"
format_stamp: "Formatted at 2024-05-03 02:14:17 on dist-test-slave-38ht"
I20240503 02:14:17.826305 7018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4334b0381be846cd801049e113b85b75"
format_stamp: "Formatted at 2024-05-03 02:14:17 on dist-test-slave-38ht"
I20240503 02:14:17.831032 7018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:14:17.834133 7033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:17.835378 7018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:14:17.835521 7018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4334b0381be846cd801049e113b85b75"
format_stamp: "Formatted at 2024-05-03 02:14:17 on dist-test-slave-38ht"
I20240503 02:14:17.835700 7018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:17.876698 7018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:17.877476 7018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:17.877727 7018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:17.878844 7018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:17.880676 7018 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:17.880777 7018 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:17.880892 7018 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:17.880964 7018 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:17.912598 7018 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.196:32849
I20240503 02:14:17.912672 7146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.196:32849 every 8 connection(s)
I20240503 02:14:17.913861 7018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:14:17.920140 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 7018
I20240503 02:14:17.920365 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:14:17.926002 7147 heartbeater.cc:347] Connected to a master server at 127.23.104.254:40161
I20240503 02:14:17.926237 7147 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:17.926959 7147 heartbeater.cc:511] Master 127.23.104.254:40161 requested a full tablet report, sending...
I20240503 02:14:17.969575 6563 ts_manager.cc:194] Registered new tserver with Master: 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849)
I20240503 02:14:17.970446 6563 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.196:38129
I20240503 02:14:17.973338 23971 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 02:14:17.988858 23971 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 02:14:17.989001 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 6755
I20240503 02:14:17.996098 6879 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:18.073258 6884 heartbeater.cc:503] Master 127.23.104.254:40161 was elected leader, sending a full tablet report...
I20240503 02:14:18.075024 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 6755
I20240503 02:14:18.092164 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 6888
I20240503 02:14:18.099754 7010 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:18.176662 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 6888
I20240503 02:14:18.973574 7147 heartbeater.cc:503] Master 127.23.104.254:40161 was elected leader, sending a full tablet report...
I20240503 02:14:21.945453 6563 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35934:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:14:21.952025 6563 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240503 02:14:21.975160 7081 tablet_service.cc:1467] Processing CreateTablet for tablet 35c2b9629a3348bda52e6a64051d59da (DEFAULT_TABLE table=rebalance_test_table_0 [id=b3827a0d78b743c4bb95ec1161954529]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:21.975513 6683 tablet_service.cc:1467] Processing CreateTablet for tablet 08cf83a491bd47cba73264964f6cf238 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b3827a0d78b743c4bb95ec1161954529]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:21.975258 7082 tablet_service.cc:1467] Processing CreateTablet for tablet e8efa40c008b491bb95c01f6f463a8b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b3827a0d78b743c4bb95ec1161954529]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:21.976176 6684 tablet_service.cc:1467] Processing CreateTablet for tablet 35c2b9629a3348bda52e6a64051d59da (DEFAULT_TABLE table=rebalance_test_table_0 [id=b3827a0d78b743c4bb95ec1161954529]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:21.976682 7081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c2b9629a3348bda52e6a64051d59da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:21.976977 6684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c2b9629a3348bda52e6a64051d59da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:21.977115 7082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8efa40c008b491bb95c01f6f463a8b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:21.977417 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08cf83a491bd47cba73264964f6cf238. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:21.978003 7080 tablet_service.cc:1467] Processing CreateTablet for tablet 08cf83a491bd47cba73264964f6cf238 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b3827a0d78b743c4bb95ec1161954529]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:21.975610 6685 tablet_service.cc:1467] Processing CreateTablet for tablet e8efa40c008b491bb95c01f6f463a8b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b3827a0d78b743c4bb95ec1161954529]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:21.978467 7080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08cf83a491bd47cba73264964f6cf238. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:21.978489 6685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8efa40c008b491bb95c01f6f463a8b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:21.993647 7171 tablet_bootstrap.cc:492] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75: Bootstrap starting.
I20240503 02:14:21.995322 7170 tablet_bootstrap.cc:492] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8: Bootstrap starting.
I20240503 02:14:21.996873 7171 tablet_bootstrap.cc:654] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:21.997668 7170 tablet_bootstrap.cc:654] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:21.997821 7171 log.cc:824] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:21.998673 7170 log.cc:824] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:22.000324 7171 tablet_bootstrap.cc:492] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75: No bootstrap required, opened a new log
I20240503 02:14:22.000537 7171 ts_tablet_manager.cc:1367] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 02:14:22.005956 7171 raft_consensus.cc:348] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.006242 7171 raft_consensus.cc:374] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:22.006350 7171 raft_consensus.cc:729] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4334b0381be846cd801049e113b85b75, State: Initialized, Role: FOLLOWER
I20240503 02:14:22.006678 7170 tablet_bootstrap.cc:492] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8: No bootstrap required, opened a new log
I20240503 02:14:22.006909 7170 ts_tablet_manager.cc:1367] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.001s
I20240503 02:14:22.006948 7171 consensus_queue.cc:260] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.010015 7171 ts_tablet_manager.cc:1398] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20240503 02:14:22.010403 7171 tablet_bootstrap.cc:492] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75: Bootstrap starting.
I20240503 02:14:22.011988 7171 tablet_bootstrap.cc:654] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:22.015332 7170 raft_consensus.cc:348] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.015748 7170 raft_consensus.cc:374] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:22.015882 7170 raft_consensus.cc:729] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Initialized, Role: FOLLOWER
I20240503 02:14:22.016574 7170 consensus_queue.cc:260] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.016853 7171 tablet_bootstrap.cc:492] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75: No bootstrap required, opened a new log
I20240503 02:14:22.016984 7171 ts_tablet_manager.cc:1367] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240503 02:14:22.017820 7171 raft_consensus.cc:348] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.017963 7171 raft_consensus.cc:374] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:22.018057 7171 raft_consensus.cc:729] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4334b0381be846cd801049e113b85b75, State: Initialized, Role: FOLLOWER
I20240503 02:14:22.018129 7170 ts_tablet_manager.cc:1398] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8: Time spent starting tablet: real 0.011s user 0.002s sys 0.009s
I20240503 02:14:22.018291 7171 consensus_queue.cc:260] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.018534 7170 tablet_bootstrap.cc:492] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8: Bootstrap starting.
I20240503 02:14:22.018862 7171 ts_tablet_manager.cc:1398] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:14:22.019089 7171 tablet_bootstrap.cc:492] T e8efa40c008b491bb95c01f6f463a8b4 P 4334b0381be846cd801049e113b85b75: Bootstrap starting.
I20240503 02:14:22.020277 7170 tablet_bootstrap.cc:654] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:22.020618 7171 tablet_bootstrap.cc:654] T e8efa40c008b491bb95c01f6f463a8b4 P 4334b0381be846cd801049e113b85b75: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:22.023195 7170 tablet_bootstrap.cc:492] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8: No bootstrap required, opened a new log
I20240503 02:14:22.023231 7171 tablet_bootstrap.cc:492] T e8efa40c008b491bb95c01f6f463a8b4 P 4334b0381be846cd801049e113b85b75: No bootstrap required, opened a new log
I20240503 02:14:22.023306 7170 ts_tablet_manager.cc:1367] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:14:22.023380 7171 ts_tablet_manager.cc:1367] T e8efa40c008b491bb95c01f6f463a8b4 P 4334b0381be846cd801049e113b85b75: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:14:22.023962 7171 raft_consensus.cc:348] T e8efa40c008b491bb95c01f6f463a8b4 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.024127 7171 raft_consensus.cc:374] T e8efa40c008b491bb95c01f6f463a8b4 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:22.024101 7170 raft_consensus.cc:348] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.024200 7171 raft_consensus.cc:729] T e8efa40c008b491bb95c01f6f463a8b4 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4334b0381be846cd801049e113b85b75, State: Initialized, Role: FOLLOWER
I20240503 02:14:22.024279 7170 raft_consensus.cc:374] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:22.024379 7170 raft_consensus.cc:729] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Initialized, Role: FOLLOWER
I20240503 02:14:22.024444 7171 consensus_queue.cc:260] T e8efa40c008b491bb95c01f6f463a8b4 P 4334b0381be846cd801049e113b85b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.024590 7170 consensus_queue.cc:260] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.025002 7170 ts_tablet_manager.cc:1398] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:14:22.024993 7171 ts_tablet_manager.cc:1398] T e8efa40c008b491bb95c01f6f463a8b4 P 4334b0381be846cd801049e113b85b75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:22.025238 7170 tablet_bootstrap.cc:492] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8: Bootstrap starting.
I20240503 02:14:22.026751 7170 tablet_bootstrap.cc:654] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:22.028548 7170 tablet_bootstrap.cc:492] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8: No bootstrap required, opened a new log
I20240503 02:14:22.028685 7170 ts_tablet_manager.cc:1367] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:14:22.029203 7170 raft_consensus.cc:348] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.029352 7170 raft_consensus.cc:374] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:22.029407 7170 raft_consensus.cc:729] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Initialized, Role: FOLLOWER
I20240503 02:14:22.029608 7170 consensus_queue.cc:260] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.030028 7170 ts_tablet_manager.cc:1398] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 02:14:22.051931 6752 tablet.cc:2369] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:14:22.131038 7176 raft_consensus.cc:482] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:22.131319 7176 raft_consensus.cc:504] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.132887 7176 leader_election.cc:288] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849)
I20240503 02:14:22.139058 7102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8efa40c008b491bb95c01f6f463a8b4" candidate_uuid: "b1ff4710a0a24719ae6a43233f261fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4334b0381be846cd801049e113b85b75" is_pre_election: true
I20240503 02:14:22.139554 7102 raft_consensus.cc:2454] T e8efa40c008b491bb95c01f6f463a8b4 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1ff4710a0a24719ae6a43233f261fb8 in term 0.
I20240503 02:14:22.140143 6638 leader_election.cc:302] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:22.140476 7176 raft_consensus.cc:2781] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:22.140620 7176 raft_consensus.cc:482] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:22.140715 7176 raft_consensus.cc:3037] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:22.142645 7176 raft_consensus.cc:504] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.143062 7176 leader_election.cc:288] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 election: Requested vote from peers 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849)
I20240503 02:14:22.143517 7102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8efa40c008b491bb95c01f6f463a8b4" candidate_uuid: "b1ff4710a0a24719ae6a43233f261fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4334b0381be846cd801049e113b85b75"
I20240503 02:14:22.143667 7102 raft_consensus.cc:3037] T e8efa40c008b491bb95c01f6f463a8b4 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:22.145659 7102 raft_consensus.cc:2454] T e8efa40c008b491bb95c01f6f463a8b4 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1ff4710a0a24719ae6a43233f261fb8 in term 1.
I20240503 02:14:22.145977 6638 leader_election.cc:302] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:22.146235 7176 raft_consensus.cc:2781] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:22.146629 7176 raft_consensus.cc:686] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Becoming Leader. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Running, Role: LEADER
I20240503 02:14:22.147101 7176 consensus_queue.cc:237] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.151139 6562 catalog_manager.cc:5484] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:22.156131 7176 raft_consensus.cc:482] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:22.156263 7176 raft_consensus.cc:504] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.156605 7176 leader_election.cc:288] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849)
I20240503 02:14:22.157042 7102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c2b9629a3348bda52e6a64051d59da" candidate_uuid: "b1ff4710a0a24719ae6a43233f261fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4334b0381be846cd801049e113b85b75" is_pre_election: true
I20240503 02:14:22.157279 7102 raft_consensus.cc:2454] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1ff4710a0a24719ae6a43233f261fb8 in term 0.
I20240503 02:14:22.157619 6638 leader_election.cc:302] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:22.157886 7176 raft_consensus.cc:2781] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:22.158004 7176 raft_consensus.cc:482] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:22.158129 7176 raft_consensus.cc:3037] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:22.159965 7176 raft_consensus.cc:504] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.160391 7176 leader_election.cc:288] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 election: Requested vote from peers 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849)
I20240503 02:14:22.160775 7102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c2b9629a3348bda52e6a64051d59da" candidate_uuid: "b1ff4710a0a24719ae6a43233f261fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4334b0381be846cd801049e113b85b75"
I20240503 02:14:22.160948 7102 raft_consensus.cc:3037] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:22.162643 7102 raft_consensus.cc:2454] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1ff4710a0a24719ae6a43233f261fb8 in term 1.
I20240503 02:14:22.162957 6638 leader_election.cc:302] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:22.163205 7176 raft_consensus.cc:2781] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:22.163368 7176 raft_consensus.cc:686] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Becoming Leader. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Running, Role: LEADER
I20240503 02:14:22.163591 7176 consensus_queue.cc:237] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.165520 6562 catalog_manager.cc:5484] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } health_report { overall_health: HEALTHY } } }
W20240503 02:14:22.168351 7148 tablet.cc:2369] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:14:22.351694 7176 raft_consensus.cc:482] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:22.351902 7176 raft_consensus.cc:504] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.352459 7176 leader_election.cc:288] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849)
I20240503 02:14:22.353026 7102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08cf83a491bd47cba73264964f6cf238" candidate_uuid: "b1ff4710a0a24719ae6a43233f261fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4334b0381be846cd801049e113b85b75" is_pre_election: true
I20240503 02:14:22.353209 7102 raft_consensus.cc:2454] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1ff4710a0a24719ae6a43233f261fb8 in term 0.
I20240503 02:14:22.353574 6638 leader_election.cc:302] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:22.353801 7176 raft_consensus.cc:2781] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:22.353911 7176 raft_consensus.cc:482] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:22.353979 7176 raft_consensus.cc:3037] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:22.355803 7176 raft_consensus.cc:504] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.356154 7176 leader_election.cc:288] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 election: Requested vote from peers 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849)
I20240503 02:14:22.356632 7102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08cf83a491bd47cba73264964f6cf238" candidate_uuid: "b1ff4710a0a24719ae6a43233f261fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4334b0381be846cd801049e113b85b75"
I20240503 02:14:22.356797 7102 raft_consensus.cc:3037] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:22.358538 7102 raft_consensus.cc:2454] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1ff4710a0a24719ae6a43233f261fb8 in term 1.
I20240503 02:14:22.358881 6638 leader_election.cc:302] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:22.359128 7176 raft_consensus.cc:2781] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:22.359305 7176 raft_consensus.cc:686] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Becoming Leader. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Running, Role: LEADER
I20240503 02:14:22.359571 7176 consensus_queue.cc:237] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:22.361922 6562 catalog_manager.cc:5484] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:22.615478 7176 consensus_queue.cc:1035] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:22.745509 7181 consensus_queue.cc:1035] T e8efa40c008b491bb95c01f6f463a8b4 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:22.754446 7185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:22.754770 7185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:22.762184 7185 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 02:14:22.900370 7194 consensus_queue.cc:1035] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:22.931396 7218 mvcc.cc:205] Tried to move back new op lower bound from 7023421288043294720 to 7023421285826322432. Current Snapshot: MvccSnapshot[applied={T|T < 7023421288043294720}]
INSERT report
rows total: 2000
time total: 175.632 ms
time per row: 0.0878162 ms
I20240503 02:14:23.087307 6562 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35934:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:14:23.088124 6562 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240503 02:14:23.095517 6685 tablet_service.cc:1467] Processing CreateTablet for tablet 3bfc771e15b44977be9a9ea9c528b977 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c3442b3ed234a8190878f87d52fec63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:23.095585 7081 tablet_service.cc:1467] Processing CreateTablet for tablet 3bfc771e15b44977be9a9ea9c528b977 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c3442b3ed234a8190878f87d52fec63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:23.096067 7081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfc771e15b44977be9a9ea9c528b977. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:23.096204 6685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfc771e15b44977be9a9ea9c528b977. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:23.096303 6684 tablet_service.cc:1467] Processing CreateTablet for tablet 02a869584fc946ad9447da0b5a2203b3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c3442b3ed234a8190878f87d52fec63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:23.096364 6683 tablet_service.cc:1467] Processing CreateTablet for tablet 18bbda38b89544048c3b6cbae8d655db (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c3442b3ed234a8190878f87d52fec63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:23.096917 6684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a869584fc946ad9447da0b5a2203b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:23.097975 7082 tablet_service.cc:1467] Processing CreateTablet for tablet 18bbda38b89544048c3b6cbae8d655db (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c3442b3ed234a8190878f87d52fec63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:23.098321 7082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18bbda38b89544048c3b6cbae8d655db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:23.102826 7080 tablet_service.cc:1467] Processing CreateTablet for tablet 02a869584fc946ad9447da0b5a2203b3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5c3442b3ed234a8190878f87d52fec63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:23.103276 7080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a869584fc946ad9447da0b5a2203b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:23.103617 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18bbda38b89544048c3b6cbae8d655db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:23.111157 7237 tablet_bootstrap.cc:492] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75: Bootstrap starting.
I20240503 02:14:23.111565 7238 tablet_bootstrap.cc:492] T 02a869584fc946ad9447da0b5a2203b3 P b1ff4710a0a24719ae6a43233f261fb8: Bootstrap starting.
I20240503 02:14:23.113113 7237 tablet_bootstrap.cc:654] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:23.113323 7238 tablet_bootstrap.cc:654] T 02a869584fc946ad9447da0b5a2203b3 P b1ff4710a0a24719ae6a43233f261fb8: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:23.115159 7237 tablet_bootstrap.cc:492] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75: No bootstrap required, opened a new log
I20240503 02:14:23.115276 7237 ts_tablet_manager.cc:1367] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:14:23.115501 7238 tablet_bootstrap.cc:492] T 02a869584fc946ad9447da0b5a2203b3 P b1ff4710a0a24719ae6a43233f261fb8: No bootstrap required, opened a new log
I20240503 02:14:23.115630 7238 ts_tablet_manager.cc:1367] T 02a869584fc946ad9447da0b5a2203b3 P b1ff4710a0a24719ae6a43233f261fb8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:14:23.115953 7237 raft_consensus.cc:348] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:23.116104 7237 raft_consensus.cc:374] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:23.116186 7237 raft_consensus.cc:729] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4334b0381be846cd801049e113b85b75, State: Initialized, Role: FOLLOWER
I20240503 02:14:23.116336 7238 raft_consensus.cc:348] T 02a869584fc946ad9447da0b5a2203b3 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:23.116386 7237 consensus_queue.cc:260] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:23.116505 7238 raft_consensus.cc:374] T 02a869584fc946ad9447da0b5a2203b3 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:23.116806 7237 ts_tablet_manager.cc:1398] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:14:23.117051 7237 tablet_bootstrap.cc:492] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75: Bootstrap starting.
I20240503 02:14:23.117106 7238 raft_consensus.cc:729] T 02a869584fc946ad9447da0b5a2203b3 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Initialized, Role: FOLLOWER
I20240503 02:14:23.118170 7237 tablet_bootstrap.cc:654] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:23.117365 7238 consensus_queue.cc:260] T 02a869584fc946ad9447da0b5a2203b3 P b1ff4710a0a24719ae6a43233f261fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:23.119089 7238 ts_tablet_manager.cc:1398] T 02a869584fc946ad9447da0b5a2203b3 P b1ff4710a0a24719ae6a43233f261fb8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:14:23.119448 7238 tablet_bootstrap.cc:492] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8: Bootstrap starting.
I20240503 02:14:23.121063 7238 tablet_bootstrap.cc:654] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:23.125099 7237 tablet_bootstrap.cc:492] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75: No bootstrap required, opened a new log
I20240503 02:14:23.125248 7237 ts_tablet_manager.cc:1367] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20240503 02:14:23.125295 7238 tablet_bootstrap.cc:492] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8: No bootstrap required, opened a new log
I20240503 02:14:23.125459 7238 ts_tablet_manager.cc:1367] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240503 02:14:23.125856 7237 raft_consensus.cc:348] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:23.126021 7237 raft_consensus.cc:374] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:23.125991 7238 raft_consensus.cc:348] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:23.126096 7237 raft_consensus.cc:729] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4334b0381be846cd801049e113b85b75, State: Initialized, Role: FOLLOWER
I20240503 02:14:23.126188 7238 raft_consensus.cc:374] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:23.126271 7238 raft_consensus.cc:729] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Initialized, Role: FOLLOWER
I20240503 02:14:23.126328 7237 consensus_queue.cc:260] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:23.126513 7238 consensus_queue.cc:260] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:23.126684 7237 ts_tablet_manager.cc:1398] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:14:23.127120 7238 ts_tablet_manager.cc:1398] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:14:23.127364 7237 tablet_bootstrap.cc:492] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75: Bootstrap starting.
I20240503 02:14:23.127424 7238 tablet_bootstrap.cc:492] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8: Bootstrap starting.
I20240503 02:14:23.128944 7237 tablet_bootstrap.cc:654] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:23.130481 7238 tablet_bootstrap.cc:654] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:23.132448 7238 tablet_bootstrap.cc:492] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8: No bootstrap required, opened a new log
I20240503 02:14:23.132479 7237 tablet_bootstrap.cc:492] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75: No bootstrap required, opened a new log
I20240503 02:14:23.132593 7238 ts_tablet_manager.cc:1367] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:14:23.132593 7237 ts_tablet_manager.cc:1367] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:14:23.133296 7238 raft_consensus.cc:348] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:23.133296 7237 raft_consensus.cc:348] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:23.133500 7237 raft_consensus.cc:374] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:23.133500 7238 raft_consensus.cc:374] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:23.133600 7237 raft_consensus.cc:729] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4334b0381be846cd801049e113b85b75, State: Initialized, Role: FOLLOWER
I20240503 02:14:23.133600 7238 raft_consensus.cc:729] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Initialized, Role: FOLLOWER
I20240503 02:14:23.133819 7238 consensus_queue.cc:260] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:23.133828 7237 consensus_queue.cc:260] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:23.134286 7238 ts_tablet_manager.cc:1398] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:14:23.134783 7237 ts_tablet_manager.cc:1398] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:23.269634 7186 raft_consensus.cc:482] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:23.269819 7186 raft_consensus.cc:504] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:23.271220 7186 leader_election.cc:288] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019)
I20240503 02:14:23.275400 7176 raft_consensus.cc:482] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:23.275566 7176 raft_consensus.cc:504] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:23.276288 7176 leader_election.cc:288] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849)
I20240503 02:14:23.276800 7101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18bbda38b89544048c3b6cbae8d655db" candidate_uuid: "b1ff4710a0a24719ae6a43233f261fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4334b0381be846cd801049e113b85b75" is_pre_election: true
I20240503 02:14:23.276993 7101 raft_consensus.cc:2454] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1ff4710a0a24719ae6a43233f261fb8 in term 0.
I20240503 02:14:23.277472 6638 leader_election.cc:302] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:23.277395 6705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfc771e15b44977be9a9ea9c528b977" candidate_uuid: "4334b0381be846cd801049e113b85b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ff4710a0a24719ae6a43233f261fb8" is_pre_election: true
I20240503 02:14:23.277732 7176 raft_consensus.cc:2781] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:23.277755 6705 raft_consensus.cc:2454] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4334b0381be846cd801049e113b85b75 in term 0.
I20240503 02:14:23.277860 7176 raft_consensus.cc:482] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:23.277958 7176 raft_consensus.cc:3037] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:23.278308 7034 leader_election.cc:302] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:23.278704 7186 raft_consensus.cc:2781] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:23.278824 7186 raft_consensus.cc:482] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:23.278914 7186 raft_consensus.cc:3037] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:23.280185 7176 raft_consensus.cc:504] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:23.280596 7176 leader_election.cc:288] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 election: Requested vote from peers 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849)
I20240503 02:14:23.280920 7186 raft_consensus.cc:504] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:23.281075 7101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18bbda38b89544048c3b6cbae8d655db" candidate_uuid: "b1ff4710a0a24719ae6a43233f261fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4334b0381be846cd801049e113b85b75"
I20240503 02:14:23.281260 7101 raft_consensus.cc:3037] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:23.281383 7186 leader_election.cc:288] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [CANDIDATE]: Term 1 election: Requested vote from peers b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019)
I20240503 02:14:23.281795 6705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfc771e15b44977be9a9ea9c528b977" candidate_uuid: "4334b0381be846cd801049e113b85b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ff4710a0a24719ae6a43233f261fb8"
I20240503 02:14:23.281942 6705 raft_consensus.cc:3037] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:23.283102 7101 raft_consensus.cc:2454] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1ff4710a0a24719ae6a43233f261fb8 in term 1.
I20240503 02:14:23.283460 6638 leader_election.cc:302] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:23.283725 7176 raft_consensus.cc:2781] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:23.283903 7176 raft_consensus.cc:686] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Becoming Leader. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Running, Role: LEADER
I20240503 02:14:23.283890 6705 raft_consensus.cc:2454] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4334b0381be846cd801049e113b85b75 in term 1.
I20240503 02:14:23.284224 7176 consensus_queue.cc:237] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:23.284350 7034 leader_election.cc:302] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:23.284619 7186 raft_consensus.cc:2781] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:23.285101 7186 raft_consensus.cc:686] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [term 1 LEADER]: Becoming Leader. State: Replica: 4334b0381be846cd801049e113b85b75, State: Running, Role: LEADER
I20240503 02:14:23.286221 7186 consensus_queue.cc:237] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:23.286760 6560 catalog_manager.cc:5484] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:23.289463 6562 catalog_manager.cc:5484] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4334b0381be846cd801049e113b85b75 (127.23.104.196). New cstate: current_term: 1 leader_uuid: "4334b0381be846cd801049e113b85b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:23.519385 7186 raft_consensus.cc:482] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:23.519585 7186 raft_consensus.cc:504] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:23.520084 7186 leader_election.cc:288] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019)
I20240503 02:14:23.520675 6705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a869584fc946ad9447da0b5a2203b3" candidate_uuid: "4334b0381be846cd801049e113b85b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ff4710a0a24719ae6a43233f261fb8" is_pre_election: true
I20240503 02:14:23.520886 6705 raft_consensus.cc:2454] T 02a869584fc946ad9447da0b5a2203b3 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4334b0381be846cd801049e113b85b75 in term 0.
I20240503 02:14:23.521283 7034 leader_election.cc:302] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:23.521570 7186 raft_consensus.cc:2781] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:23.521663 7186 raft_consensus.cc:482] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:23.521770 7186 raft_consensus.cc:3037] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:23.523480 7186 raft_consensus.cc:504] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:23.523929 7186 leader_election.cc:288] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [CANDIDATE]: Term 1 election: Requested vote from peers b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019)
I20240503 02:14:23.524390 6705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a869584fc946ad9447da0b5a2203b3" candidate_uuid: "4334b0381be846cd801049e113b85b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ff4710a0a24719ae6a43233f261fb8"
I20240503 02:14:23.524570 6705 raft_consensus.cc:3037] T 02a869584fc946ad9447da0b5a2203b3 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:23.526257 6705 raft_consensus.cc:2454] T 02a869584fc946ad9447da0b5a2203b3 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4334b0381be846cd801049e113b85b75 in term 1.
I20240503 02:14:23.526618 7034 leader_election.cc:302] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:23.526883 7186 raft_consensus.cc:2781] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:23.527031 7186 raft_consensus.cc:686] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [term 1 LEADER]: Becoming Leader. State: Replica: 4334b0381be846cd801049e113b85b75, State: Running, Role: LEADER
I20240503 02:14:23.527257 7186 consensus_queue.cc:237] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:23.529438 6562 catalog_manager.cc:5484] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4334b0381be846cd801049e113b85b75 (127.23.104.196). New cstate: current_term: 1 leader_uuid: "4334b0381be846cd801049e113b85b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:23.661921 7176 consensus_queue.cc:1035] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:23.667296 7246 consensus_queue.cc:1035] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:23.900513 7250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:23.900835 7250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:23.908258 7250 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:14:23.950392 7186 consensus_queue.cc:1035] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 606.853 ms
time per row: 0.303427 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:14:25.562764 6562 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35934:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:14:25.563423 6562 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240503 02:14:25.571218 6684 tablet_service.cc:1467] Processing CreateTablet for tablet d0455e93fffe45b0a4253ff7d4bc7e14 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c07dd68b46ac4aaf865ab28b1d7bf701]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:25.571669 6683 tablet_service.cc:1467] Processing CreateTablet for tablet 3bd8b03234ab4d6cb280c42bfc4e3f38 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c07dd68b46ac4aaf865ab28b1d7bf701]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:25.571693 7080 tablet_service.cc:1467] Processing CreateTablet for tablet d0455e93fffe45b0a4253ff7d4bc7e14 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c07dd68b46ac4aaf865ab28b1d7bf701]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:25.571688 6684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0455e93fffe45b0a4253ff7d4bc7e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:25.572158 7080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0455e93fffe45b0a4253ff7d4bc7e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:25.572208 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bd8b03234ab4d6cb280c42bfc4e3f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:25.572489 7082 tablet_service.cc:1467] Processing CreateTablet for tablet d6608b0fa1f94f61b6e3df7e276f3505 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c07dd68b46ac4aaf865ab28b1d7bf701]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:25.572825 7082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6608b0fa1f94f61b6e3df7e276f3505. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:25.576396 6685 tablet_service.cc:1467] Processing CreateTablet for tablet d6608b0fa1f94f61b6e3df7e276f3505 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c07dd68b46ac4aaf865ab28b1d7bf701]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:25.576809 6685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6608b0fa1f94f61b6e3df7e276f3505. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:25.576961 7081 tablet_service.cc:1467] Processing CreateTablet for tablet 3bd8b03234ab4d6cb280c42bfc4e3f38 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c07dd68b46ac4aaf865ab28b1d7bf701]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:25.577277 7081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bd8b03234ab4d6cb280c42bfc4e3f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:25.583541 7309 tablet_bootstrap.cc:492] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75: Bootstrap starting.
I20240503 02:14:25.585202 7309 tablet_bootstrap.cc:654] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:25.587486 7308 tablet_bootstrap.cc:492] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8: Bootstrap starting.
I20240503 02:14:25.587749 7309 tablet_bootstrap.cc:492] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75: No bootstrap required, opened a new log
I20240503 02:14:25.587909 7309 ts_tablet_manager.cc:1367] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:14:25.588573 7309 raft_consensus.cc:348] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:25.588799 7309 raft_consensus.cc:374] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:25.588944 7309 raft_consensus.cc:729] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4334b0381be846cd801049e113b85b75, State: Initialized, Role: FOLLOWER
I20240503 02:14:25.589161 7309 consensus_queue.cc:260] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:25.589293 7308 tablet_bootstrap.cc:654] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:25.589643 7309 ts_tablet_manager.cc:1398] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:14:25.589942 7309 tablet_bootstrap.cc:492] T d6608b0fa1f94f61b6e3df7e276f3505 P 4334b0381be846cd801049e113b85b75: Bootstrap starting.
I20240503 02:14:25.591070 7308 tablet_bootstrap.cc:492] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8: No bootstrap required, opened a new log
I20240503 02:14:25.591205 7308 ts_tablet_manager.cc:1367] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:14:25.591325 7309 tablet_bootstrap.cc:654] T d6608b0fa1f94f61b6e3df7e276f3505 P 4334b0381be846cd801049e113b85b75: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:25.591979 7308 raft_consensus.cc:348] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:25.592224 7308 raft_consensus.cc:374] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:25.592346 7308 raft_consensus.cc:729] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Initialized, Role: FOLLOWER
I20240503 02:14:25.592687 7308 consensus_queue.cc:260] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:25.593163 7308 ts_tablet_manager.cc:1398] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:25.593469 7308 tablet_bootstrap.cc:492] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8: Bootstrap starting.
I20240503 02:14:25.594761 7308 tablet_bootstrap.cc:654] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:25.596210 7176 raft_consensus.cc:482] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:25.596367 7176 raft_consensus.cc:504] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:25.596483 7308 tablet_bootstrap.cc:492] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8: No bootstrap required, opened a new log
I20240503 02:14:25.596629 7308 ts_tablet_manager.cc:1367] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:14:25.596763 7309 tablet_bootstrap.cc:492] T d6608b0fa1f94f61b6e3df7e276f3505 P 4334b0381be846cd801049e113b85b75: No bootstrap required, opened a new log
I20240503 02:14:25.596892 7176 leader_election.cc:288] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849)
I20240503 02:14:25.596930 7309 ts_tablet_manager.cc:1367] T d6608b0fa1f94f61b6e3df7e276f3505 P 4334b0381be846cd801049e113b85b75: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240503 02:14:25.597362 7308 raft_consensus.cc:348] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:25.597515 7308 raft_consensus.cc:374] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:25.597594 7308 raft_consensus.cc:729] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Initialized, Role: FOLLOWER
I20240503 02:14:25.597563 7101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0455e93fffe45b0a4253ff7d4bc7e14" candidate_uuid: "b1ff4710a0a24719ae6a43233f261fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4334b0381be846cd801049e113b85b75" is_pre_election: true
I20240503 02:14:25.597810 7101 raft_consensus.cc:2454] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1ff4710a0a24719ae6a43233f261fb8 in term 0.
I20240503 02:14:25.597813 7308 consensus_queue.cc:260] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:25.597782 7309 raft_consensus.cc:348] T d6608b0fa1f94f61b6e3df7e276f3505 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:25.597966 7309 raft_consensus.cc:374] T d6608b0fa1f94f61b6e3df7e276f3505 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:25.598038 7309 raft_consensus.cc:729] T d6608b0fa1f94f61b6e3df7e276f3505 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4334b0381be846cd801049e113b85b75, State: Initialized, Role: FOLLOWER
I20240503 02:14:25.598158 7308 ts_tablet_manager.cc:1398] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:14:25.598376 7308 tablet_bootstrap.cc:492] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8: Bootstrap starting.
I20240503 02:14:25.598258 7309 consensus_queue.cc:260] T d6608b0fa1f94f61b6e3df7e276f3505 P 4334b0381be846cd801049e113b85b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:25.598524 6638 leader_election.cc:302] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:25.598742 7309 ts_tablet_manager.cc:1398] T d6608b0fa1f94f61b6e3df7e276f3505 P 4334b0381be846cd801049e113b85b75: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:14:25.598860 7303 raft_consensus.cc:2781] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:25.598978 7303 raft_consensus.cc:482] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:25.599115 7303 raft_consensus.cc:3037] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:25.599936 7308 tablet_bootstrap.cc:654] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:25.602509 7303 raft_consensus.cc:504] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:25.602574 7309 tablet_bootstrap.cc:492] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75: Bootstrap starting.
I20240503 02:14:25.603130 7303 leader_election.cc:288] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 election: Requested vote from peers 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849)
I20240503 02:14:25.603559 7101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0455e93fffe45b0a4253ff7d4bc7e14" candidate_uuid: "b1ff4710a0a24719ae6a43233f261fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4334b0381be846cd801049e113b85b75"
I20240503 02:14:25.603778 7101 raft_consensus.cc:3037] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:25.604228 7309 tablet_bootstrap.cc:654] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:25.604633 7308 tablet_bootstrap.cc:492] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8: No bootstrap required, opened a new log
I20240503 02:14:25.604722 7308 ts_tablet_manager.cc:1367] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:14:25.605211 7308 raft_consensus.cc:348] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:25.605383 7308 raft_consensus.cc:374] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:25.605479 7308 raft_consensus.cc:729] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Initialized, Role: FOLLOWER
I20240503 02:14:25.605715 7308 consensus_queue.cc:260] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:25.606155 7308 ts_tablet_manager.cc:1398] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:14:25.606185 7309 tablet_bootstrap.cc:492] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75: No bootstrap required, opened a new log
I20240503 02:14:25.606305 7309 ts_tablet_manager.cc:1367] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:14:25.606695 7101 raft_consensus.cc:2454] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1ff4710a0a24719ae6a43233f261fb8 in term 1.
I20240503 02:14:25.607295 6638 leader_election.cc:302] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:25.607587 7303 raft_consensus.cc:2781] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:25.607756 7303 raft_consensus.cc:686] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Becoming Leader. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Running, Role: LEADER
I20240503 02:14:25.607988 7303 consensus_queue.cc:237] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } }
I20240503 02:14:25.608312 7309 raft_consensus.cc:348] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:25.608479 7309 raft_consensus.cc:374] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:25.608542 7309 raft_consensus.cc:729] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4334b0381be846cd801049e113b85b75, State: Initialized, Role: FOLLOWER
I20240503 02:14:25.608743 7309 consensus_queue.cc:260] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:25.609251 7309 ts_tablet_manager.cc:1398] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:14:25.610203 6562 catalog_manager.cc:5484] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:25.689023 7297 raft_consensus.cc:482] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:25.689213 7297 raft_consensus.cc:504] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:25.689731 7297 leader_election.cc:288] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019)
I20240503 02:14:25.690253 6705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bd8b03234ab4d6cb280c42bfc4e3f38" candidate_uuid: "4334b0381be846cd801049e113b85b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ff4710a0a24719ae6a43233f261fb8" is_pre_election: true
I20240503 02:14:25.690450 6705 raft_consensus.cc:2454] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4334b0381be846cd801049e113b85b75 in term 0.
I20240503 02:14:25.690867 7034 leader_election.cc:302] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:25.691102 7297 raft_consensus.cc:2781] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:25.691217 7297 raft_consensus.cc:482] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:25.691282 7297 raft_consensus.cc:3037] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:25.693239 7297 raft_consensus.cc:504] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:25.693598 7297 leader_election.cc:288] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [CANDIDATE]: Term 1 election: Requested vote from peers b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019)
I20240503 02:14:25.694031 6705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bd8b03234ab4d6cb280c42bfc4e3f38" candidate_uuid: "4334b0381be846cd801049e113b85b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ff4710a0a24719ae6a43233f261fb8"
I20240503 02:14:25.694200 6705 raft_consensus.cc:3037] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:25.696128 6705 raft_consensus.cc:2454] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4334b0381be846cd801049e113b85b75 in term 1.
I20240503 02:14:25.696482 7034 leader_election.cc:302] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:25.696770 7297 raft_consensus.cc:2781] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:25.697134 7297 raft_consensus.cc:686] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [term 1 LEADER]: Becoming Leader. State: Replica: 4334b0381be846cd801049e113b85b75, State: Running, Role: LEADER
I20240503 02:14:25.697432 7297 consensus_queue.cc:237] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:25.699612 6562 catalog_manager.cc:5484] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4334b0381be846cd801049e113b85b75 (127.23.104.196). New cstate: current_term: 1 leader_uuid: "4334b0381be846cd801049e113b85b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:25.833053 7303 raft_consensus.cc:482] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:25.833210 7303 raft_consensus.cc:504] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:25.833732 7303 leader_election.cc:288] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849)
I20240503 02:14:25.834265 7101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6608b0fa1f94f61b6e3df7e276f3505" candidate_uuid: "b1ff4710a0a24719ae6a43233f261fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4334b0381be846cd801049e113b85b75" is_pre_election: true
I20240503 02:14:25.834491 7101 raft_consensus.cc:2454] T d6608b0fa1f94f61b6e3df7e276f3505 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1ff4710a0a24719ae6a43233f261fb8 in term 0.
I20240503 02:14:25.834898 6638 leader_election.cc:302] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:25.835211 7303 raft_consensus.cc:2781] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:25.835314 7303 raft_consensus.cc:482] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:25.835407 7303 raft_consensus.cc:3037] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:25.837397 7303 raft_consensus.cc:504] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:25.837785 7303 leader_election.cc:288] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 election: Requested vote from peers 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849)
I20240503 02:14:25.838218 7101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6608b0fa1f94f61b6e3df7e276f3505" candidate_uuid: "b1ff4710a0a24719ae6a43233f261fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4334b0381be846cd801049e113b85b75"
I20240503 02:14:25.838371 7101 raft_consensus.cc:3037] T d6608b0fa1f94f61b6e3df7e276f3505 P 4334b0381be846cd801049e113b85b75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:25.840298 7101 raft_consensus.cc:2454] T d6608b0fa1f94f61b6e3df7e276f3505 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1ff4710a0a24719ae6a43233f261fb8 in term 1.
I20240503 02:14:25.840667 6638 leader_election.cc:302] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:25.840945 7303 raft_consensus.cc:2781] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:25.841064 7303 raft_consensus.cc:686] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Becoming Leader. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Running, Role: LEADER
I20240503 02:14:25.841298 7303 consensus_queue.cc:237] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } }
I20240503 02:14:25.843443 6562 catalog_manager.cc:5484] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193). New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:26.062186 7303 consensus_queue.cc:1035] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:26.121232 7297 consensus_queue.cc:1035] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:26.203758 7320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:26.204121 7320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:26.211195 7320 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 02:14:26.301119 7303 consensus_queue.cc:1035] T d6608b0fa1f94f61b6e3df7e276f3505 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 141.583 ms
time per row: 0.0707917 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:14:27.389454 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.194:33215
--local_ip_for_outbound_sockets=127.23.104.194
--tserver_master_addrs=127.23.104.254:40161
--webserver_port=35565
--webserver_interface=127.23.104.194
--builtin_ntp_servers=127.23.104.212:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:14:27.714454 7367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:27.714713 7367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:27.714797 7367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:27.722721 7367 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:27.722870 7367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:27.723052 7367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.194
I20240503 02:14:27.736833 7367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.194:33215
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.23.104.194
--webserver_port=35565
--tserver_master_addrs=127.23.104.254:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.194
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:27.738248 7367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:27.740173 7367 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:27.753342 7374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:27.753834 7376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:27.754330 7373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:27.754966 7367 server_base.cc:1034] running on GCE node
I20240503 02:14:27.755483 7367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:27.756386 7367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:27.757575 7367 hybrid_clock.cc:648] HybridClock initialized: now 1714702467757525 us; error 41 us; skew 500 ppm
I20240503 02:14:27.757858 7367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:27.759976 7367 webserver.cc:460] Webserver started at http://127.23.104.194:35565/ using document root <none> and password file <none>
I20240503 02:14:27.760680 7367 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:27.760809 7367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:27.765550 7367 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:14:27.767992 7382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:27.769167 7367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:14:27.769335 7367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ee60d5902a7044f9b55e6702b4d8a29a"
format_stamp: "Formatted at 2024-05-03 02:14:16 on dist-test-slave-38ht"
I20240503 02:14:27.770080 7367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:27.798934 7367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:27.799726 7367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:27.800019 7367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:27.800906 7367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:27.802322 7367 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:27.802402 7367 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:27.802532 7367 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:27.802589 7367 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:27.833074 7367 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.194:33215
I20240503 02:14:27.833125 7495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.194:33215 every 8 connection(s)
I20240503 02:14:27.834436 7367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:14:27.838238 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 7367
I20240503 02:14:27.839012 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.195:32971
--local_ip_for_outbound_sockets=127.23.104.195
--tserver_master_addrs=127.23.104.254:40161
--webserver_port=46287
--webserver_interface=127.23.104.195
--builtin_ntp_servers=127.23.104.212:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:27.848119 7496 heartbeater.cc:347] Connected to a master server at 127.23.104.254:40161
I20240503 02:14:27.848380 7496 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:27.849107 7496 heartbeater.cc:511] Master 127.23.104.254:40161 requested a full tablet report, sending...
I20240503 02:14:27.850576 6555 ts_manager.cc:194] Re-registered known tserver with Master: ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194:33215)
I20240503 02:14:27.851697 6555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.194:41225
W20240503 02:14:28.173023 7500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:28.173255 7500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:28.173369 7500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:28.180063 7500 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:28.180171 7500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:28.180344 7500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.195
I20240503 02:14:28.188586 7500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.195:32971
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.23.104.195
--webserver_port=46287
--tserver_master_addrs=127.23.104.254:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.195
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:28.189759 7500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:28.191381 7500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:28.203094 7506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:28.203564 7508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:28.204064 7505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:28.204695 7500 server_base.cc:1034] running on GCE node
I20240503 02:14:28.205231 7500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:28.206059 7500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:28.207262 7500 hybrid_clock.cc:648] HybridClock initialized: now 1714702468207222 us; error 37 us; skew 500 ppm
I20240503 02:14:28.207590 7500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:28.209717 7500 webserver.cc:460] Webserver started at http://127.23.104.195:46287/ using document root <none> and password file <none>
I20240503 02:14:28.210381 7500 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:28.210512 7500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:28.215299 7500 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:14:28.217790 7514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:28.219003 7500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:28.219151 7500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4fbc402737f64c5b9297ff0f7e22044e"
format_stamp: "Formatted at 2024-05-03 02:14:17 on dist-test-slave-38ht"
I20240503 02:14:28.219961 7500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:28.251122 7500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:28.251947 7500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:28.252275 7500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:28.253203 7500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:28.254616 7500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:28.254709 7500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:28.254832 7500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:28.254894 7500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:28.285038 7500 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.195:32971
I20240503 02:14:28.285115 7626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.195:32971 every 8 connection(s)
I20240503 02:14:28.286442 7500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:14:28.290565 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 7500
I20240503 02:14:28.298282 7627 heartbeater.cc:347] Connected to a master server at 127.23.104.254:40161
I20240503 02:14:28.298547 7627 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:28.299340 7627 heartbeater.cc:511] Master 127.23.104.254:40161 requested a full tablet report, sending...
I20240503 02:14:28.300818 6555 ts_manager.cc:194] Re-registered known tserver with Master: 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195:32971)
I20240503 02:14:28.301872 6555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.195:51605
I20240503 02:14:28.766649 7562 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:28.768849 7082 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:28.772789 6685 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:14:28.775741 7431 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:28.853958 7496 heartbeater.cc:503] Master 127.23.104.254:40161 was elected leader, sending a full tablet report...
I20240503 02:14:28.940838 7562 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:28.942521 7082 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:28.949313 6685 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:14:28.950030 7431 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:29.106937 7082 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:29.121788 7431 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:29.122604 7562 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:29.122817 6685 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:14:29.266487 6685 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:14:29.267397 7082 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:29.268299 7562 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:29.276265 7431 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:29.306154 7627 heartbeater.cc:503] Master 127.23.104.254:40161 was elected leader, sending a full tablet report...
I20240503 02:14:29.344213 6705 consensus_queue.cc:237] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } }
I20240503 02:14:29.346536 7102 raft_consensus.cc:1261] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Refusing update from remote peer b1ff4710a0a24719ae6a43233f261fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:29.347121 7767 consensus_queue.cc:1035] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:29.349840 7303 raft_consensus.cc:2932] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } } }
I20240503 02:14:29.351343 7102 raft_consensus.cc:2932] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } } }
W20240503 02:14:29.354163 6640 consensus_peers.cc:485] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 -> Peer 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195:32971): Couldn't send request to peer 4fbc402737f64c5b9297ff0f7e22044e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0455e93fffe45b0a4253ff7d4bc7e14. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:29.355592 6554 catalog_manager.cc:5484] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 reported cstate change: config changed from index -1 to 4, NON_VOTER 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) added. New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:29.367338 7102 consensus_queue.cc:237] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } }
I20240503 02:14:29.369570 6705 raft_consensus.cc:1261] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Refusing update from remote peer 4334b0381be846cd801049e113b85b75: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:29.370221 7317 consensus_queue.cc:1035] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:29.372897 7780 raft_consensus.cc:2932] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } } }
I20240503 02:14:29.373550 6705 raft_consensus.cc:2932] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } } }
I20240503 02:14:29.378351 6558 catalog_manager.cc:5484] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 reported cstate change: config changed from index -1 to 4, NON_VOTER ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) added. New cstate: current_term: 1 leader_uuid: "4334b0381be846cd801049e113b85b75" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:14:29.380699 7037 consensus_peers.cc:485] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 -> Peer ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194:33215): Couldn't send request to peer ee60d5902a7044f9b55e6702b4d8a29a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bfc771e15b44977be9a9ea9c528b977. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:29.389075 6705 consensus_queue.cc:237] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } }
I20240503 02:14:29.390672 7102 raft_consensus.cc:1261] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Refusing update from remote peer b1ff4710a0a24719ae6a43233f261fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:14:29.390937 6640 consensus_peers.cc:485] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 -> Peer 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195:32971): Couldn't send request to peer 4fbc402737f64c5b9297ff0f7e22044e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18bbda38b89544048c3b6cbae8d655db. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:29.391187 7303 consensus_queue.cc:1035] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:29.393435 7767 raft_consensus.cc:2932] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } } }
I20240503 02:14:29.393990 7102 raft_consensus.cc:2932] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } } }
I20240503 02:14:29.398286 6554 catalog_manager.cc:5484] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 reported cstate change: config changed from index -1 to 4, NON_VOTER 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) added. New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:29.409214 7102 consensus_queue.cc:237] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } }
I20240503 02:14:29.410842 6705 raft_consensus.cc:1261] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Refusing update from remote peer 4334b0381be846cd801049e113b85b75: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:14:29.411119 7037 consensus_peers.cc:485] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 -> Peer ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194:33215): Couldn't send request to peer ee60d5902a7044f9b55e6702b4d8a29a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bd8b03234ab4d6cb280c42bfc4e3f38. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:29.411347 7780 consensus_queue.cc:1035] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:29.413507 7317 raft_consensus.cc:2932] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } } }
I20240503 02:14:29.414098 6705 raft_consensus.cc:2932] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } } }
I20240503 02:14:29.417229 6558 catalog_manager.cc:5484] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 reported cstate change: config changed from index -1 to 4, NON_VOTER ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) added. New cstate: current_term: 1 leader_uuid: "4334b0381be846cd801049e113b85b75" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:29.427584 6705 consensus_queue.cc:237] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } }
I20240503 02:14:29.429998 7102 raft_consensus.cc:1261] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Refusing update from remote peer b1ff4710a0a24719ae6a43233f261fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:29.430471 7767 consensus_queue.cc:1035] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:29.432935 7303 raft_consensus.cc:2932] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } } }
I20240503 02:14:29.433408 7102 raft_consensus.cc:2932] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } } }
I20240503 02:14:29.438308 6554 catalog_manager.cc:5484] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 reported cstate change: config changed from index -1 to 4, NON_VOTER ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) added. New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:14:29.439764 6641 consensus_peers.cc:485] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 -> Peer ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194:33215): Couldn't send request to peer ee60d5902a7044f9b55e6702b4d8a29a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08cf83a491bd47cba73264964f6cf238. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:29.448683 6705 consensus_queue.cc:237] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } }
I20240503 02:14:29.450325 7102 raft_consensus.cc:1261] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Refusing update from remote peer b1ff4710a0a24719ae6a43233f261fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:14:29.450626 6640 consensus_peers.cc:485] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 -> Peer 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195:32971): Couldn't send request to peer 4fbc402737f64c5b9297ff0f7e22044e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35c2b9629a3348bda52e6a64051d59da. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:29.450834 7303 consensus_queue.cc:1035] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:29.453127 7767 raft_consensus.cc:2932] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } } }
I20240503 02:14:29.453696 7102 raft_consensus.cc:2932] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } } }
I20240503 02:14:29.458207 6555 catalog_manager.cc:5484] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 reported cstate change: config changed from index -1 to 4, NON_VOTER 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) added. New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:29.798307 7872 ts_tablet_manager.cc:902] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a: Initiating tablet copy from peer 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849)
I20240503 02:14:29.799960 7872 tablet_copy_client.cc:308] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:32849
I20240503 02:14:29.805994 7122 tablet_copy_service.cc:140] P 4334b0381be846cd801049e113b85b75: Received BeginTabletCopySession request for tablet 3bfc771e15b44977be9a9ea9c528b977 from peer ee60d5902a7044f9b55e6702b4d8a29a ({username='slave'} at 127.23.104.194:56197)
I20240503 02:14:29.806185 7122 tablet_copy_service.cc:161] P 4334b0381be846cd801049e113b85b75: Beginning new tablet copy session on tablet 3bfc771e15b44977be9a9ea9c528b977 from peer ee60d5902a7044f9b55e6702b4d8a29a at {username='slave'} at 127.23.104.194:56197: session id = ee60d5902a7044f9b55e6702b4d8a29a-3bfc771e15b44977be9a9ea9c528b977
I20240503 02:14:29.808643 7122 tablet_copy_source_session.cc:205] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:29.812726 7872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfc771e15b44977be9a9ea9c528b977. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:29.817736 7878 ts_tablet_manager.cc:902] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a: Initiating tablet copy from peer 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849)
I20240503 02:14:29.819116 7878 tablet_copy_client.cc:308] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:32849
I20240503 02:14:29.819769 7122 tablet_copy_service.cc:140] P 4334b0381be846cd801049e113b85b75: Received BeginTabletCopySession request for tablet 3bd8b03234ab4d6cb280c42bfc4e3f38 from peer ee60d5902a7044f9b55e6702b4d8a29a ({username='slave'} at 127.23.104.194:56197)
I20240503 02:14:29.819958 7122 tablet_copy_service.cc:161] P 4334b0381be846cd801049e113b85b75: Beginning new tablet copy session on tablet 3bd8b03234ab4d6cb280c42bfc4e3f38 from peer ee60d5902a7044f9b55e6702b4d8a29a at {username='slave'} at 127.23.104.194:56197: session id = ee60d5902a7044f9b55e6702b4d8a29a-3bd8b03234ab4d6cb280c42bfc4e3f38
I20240503 02:14:29.821738 7122 tablet_copy_source_session.cc:205] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:29.822840 7878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bd8b03234ab4d6cb280c42bfc4e3f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:29.823361 7872 tablet_copy_client.cc:783] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:29.823720 7872 tablet_copy_client.cc:647] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:29.825868 7883 ts_tablet_manager.cc:902] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a: Initiating tablet copy from peer b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019)
I20240503 02:14:29.826145 7872 tablet_copy_client.cc:523] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:29.828912 7878 tablet_copy_client.cc:783] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:29.829211 7878 tablet_copy_client.cc:647] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:29.830485 7872 tablet_bootstrap.cc:492] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a: Bootstrap starting.
I20240503 02:14:29.830739 7883 tablet_copy_client.cc:308] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.193:36019
I20240503 02:14:29.831008 7878 tablet_copy_client.cc:523] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:29.834231 7878 tablet_bootstrap.cc:492] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a: Bootstrap starting.
I20240503 02:14:29.838727 6726 tablet_copy_service.cc:140] P b1ff4710a0a24719ae6a43233f261fb8: Received BeginTabletCopySession request for tablet 08cf83a491bd47cba73264964f6cf238 from peer ee60d5902a7044f9b55e6702b4d8a29a ({username='slave'} at 127.23.104.194:33381)
I20240503 02:14:29.838977 6726 tablet_copy_service.cc:161] P b1ff4710a0a24719ae6a43233f261fb8: Beginning new tablet copy session on tablet 08cf83a491bd47cba73264964f6cf238 from peer ee60d5902a7044f9b55e6702b4d8a29a at {username='slave'} at 127.23.104.194:33381: session id = ee60d5902a7044f9b55e6702b4d8a29a-08cf83a491bd47cba73264964f6cf238
I20240503 02:14:29.842797 6726 tablet_copy_source_session.cc:205] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:29.843814 7872 log.cc:824] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:29.845219 7883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08cf83a491bd47cba73264964f6cf238. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:29.853754 7883 tablet_copy_client.cc:783] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:29.854077 7883 tablet_copy_client.cc:647] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:29.856832 7883 tablet_copy_client.cc:523] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:29.859210 7883 tablet_bootstrap.cc:492] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a: Bootstrap starting.
I20240503 02:14:29.882328 7878 tablet_bootstrap.cc:492] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:29.882910 7878 tablet_bootstrap.cc:492] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a: Bootstrap complete.
I20240503 02:14:29.883443 7878 ts_tablet_manager.cc:1367] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a: Time spent bootstrapping tablet: real 0.049s user 0.040s sys 0.009s
I20240503 02:14:29.892060 7878 raft_consensus.cc:348] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } }
I20240503 02:14:29.892800 7878 raft_consensus.cc:729] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee60d5902a7044f9b55e6702b4d8a29a, State: Initialized, Role: LEARNER
I20240503 02:14:29.893633 7878 consensus_queue.cc:260] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } }
I20240503 02:14:29.895213 7891 ts_tablet_manager.cc:902] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e: Initiating tablet copy from peer b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019)
I20240503 02:14:29.898576 7872 tablet_bootstrap.cc:492] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:29.899096 7872 tablet_bootstrap.cc:492] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a: Bootstrap complete.
I20240503 02:14:29.899425 7872 ts_tablet_manager.cc:1367] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a: Time spent bootstrapping tablet: real 0.069s user 0.054s sys 0.011s
I20240503 02:14:29.900494 7872 raft_consensus.cc:348] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } }
I20240503 02:14:29.900776 7872 raft_consensus.cc:729] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee60d5902a7044f9b55e6702b4d8a29a, State: Initialized, Role: LEARNER
I20240503 02:14:29.901010 7872 consensus_queue.cc:260] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } }
I20240503 02:14:29.905575 7878 ts_tablet_manager.cc:1398] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a: Time spent starting tablet: real 0.022s user 0.009s sys 0.012s
I20240503 02:14:29.908835 7872 ts_tablet_manager.cc:1398] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a: Time spent starting tablet: real 0.009s user 0.000s sys 0.009s
I20240503 02:14:29.909829 7891 tablet_copy_client.cc:308] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.193:36019
I20240503 02:14:29.909907 7122 tablet_copy_service.cc:342] P 4334b0381be846cd801049e113b85b75: Request end of tablet copy session ee60d5902a7044f9b55e6702b4d8a29a-3bfc771e15b44977be9a9ea9c528b977 received from {username='slave'} at 127.23.104.194:56197
I20240503 02:14:29.910059 7122 tablet_copy_service.cc:433] P 4334b0381be846cd801049e113b85b75: ending tablet copy session ee60d5902a7044f9b55e6702b4d8a29a-3bfc771e15b44977be9a9ea9c528b977 on tablet 3bfc771e15b44977be9a9ea9c528b977 with peer ee60d5902a7044f9b55e6702b4d8a29a
I20240503 02:14:29.910952 7121 tablet_copy_service.cc:342] P 4334b0381be846cd801049e113b85b75: Request end of tablet copy session ee60d5902a7044f9b55e6702b4d8a29a-3bd8b03234ab4d6cb280c42bfc4e3f38 received from {username='slave'} at 127.23.104.194:56197
I20240503 02:14:29.911103 7121 tablet_copy_service.cc:433] P 4334b0381be846cd801049e113b85b75: ending tablet copy session ee60d5902a7044f9b55e6702b4d8a29a-3bd8b03234ab4d6cb280c42bfc4e3f38 on tablet 3bd8b03234ab4d6cb280c42bfc4e3f38 with peer ee60d5902a7044f9b55e6702b4d8a29a
I20240503 02:14:29.916850 7883 tablet_bootstrap.cc:492] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:29.917277 7883 tablet_bootstrap.cc:492] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a: Bootstrap complete.
I20240503 02:14:29.917572 7883 ts_tablet_manager.cc:1367] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a: Time spent bootstrapping tablet: real 0.058s user 0.055s sys 0.000s
I20240503 02:14:29.918371 7883 raft_consensus.cc:348] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } }
I20240503 02:14:29.918620 7883 raft_consensus.cc:729] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee60d5902a7044f9b55e6702b4d8a29a, State: Initialized, Role: LEARNER
I20240503 02:14:29.918813 7883 consensus_queue.cc:260] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: true } }
I20240503 02:14:29.919481 7883 ts_tablet_manager.cc:1398] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:14:29.919571 6726 tablet_copy_service.cc:140] P b1ff4710a0a24719ae6a43233f261fb8: Received BeginTabletCopySession request for tablet 35c2b9629a3348bda52e6a64051d59da from peer 4fbc402737f64c5b9297ff0f7e22044e ({username='slave'} at 127.23.104.195:39709)
I20240503 02:14:29.919734 6726 tablet_copy_service.cc:161] P b1ff4710a0a24719ae6a43233f261fb8: Beginning new tablet copy session on tablet 35c2b9629a3348bda52e6a64051d59da from peer 4fbc402737f64c5b9297ff0f7e22044e at {username='slave'} at 127.23.104.195:39709: session id = 4fbc402737f64c5b9297ff0f7e22044e-35c2b9629a3348bda52e6a64051d59da
I20240503 02:14:29.920331 6725 tablet_copy_service.cc:342] P b1ff4710a0a24719ae6a43233f261fb8: Request end of tablet copy session ee60d5902a7044f9b55e6702b4d8a29a-08cf83a491bd47cba73264964f6cf238 received from {username='slave'} at 127.23.104.194:33381
I20240503 02:14:29.920495 6725 tablet_copy_service.cc:433] P b1ff4710a0a24719ae6a43233f261fb8: ending tablet copy session ee60d5902a7044f9b55e6702b4d8a29a-08cf83a491bd47cba73264964f6cf238 on tablet 08cf83a491bd47cba73264964f6cf238 with peer ee60d5902a7044f9b55e6702b4d8a29a
I20240503 02:14:29.922830 6726 tablet_copy_source_session.cc:205] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:29.927167 7891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c2b9629a3348bda52e6a64051d59da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:29.935814 7899 ts_tablet_manager.cc:902] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e: Initiating tablet copy from peer b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019)
I20240503 02:14:29.937481 7899 tablet_copy_client.cc:308] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.193:36019
I20240503 02:14:29.938217 6726 tablet_copy_service.cc:140] P b1ff4710a0a24719ae6a43233f261fb8: Received BeginTabletCopySession request for tablet d0455e93fffe45b0a4253ff7d4bc7e14 from peer 4fbc402737f64c5b9297ff0f7e22044e ({username='slave'} at 127.23.104.195:39709)
I20240503 02:14:29.938390 6726 tablet_copy_service.cc:161] P b1ff4710a0a24719ae6a43233f261fb8: Beginning new tablet copy session on tablet d0455e93fffe45b0a4253ff7d4bc7e14 from peer 4fbc402737f64c5b9297ff0f7e22044e at {username='slave'} at 127.23.104.195:39709: session id = 4fbc402737f64c5b9297ff0f7e22044e-d0455e93fffe45b0a4253ff7d4bc7e14
I20240503 02:14:29.938556 7891 tablet_copy_client.cc:783] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:29.938874 7891 tablet_copy_client.cc:647] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:29.941143 7891 tablet_copy_client.cc:523] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:29.941383 6726 tablet_copy_source_session.cc:205] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:29.942595 7899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0455e93fffe45b0a4253ff7d4bc7e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:29.945179 7891 tablet_bootstrap.cc:492] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e: Bootstrap starting.
I20240503 02:14:29.948486 7899 tablet_copy_client.cc:783] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:29.948756 7899 tablet_copy_client.cc:647] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:29.950502 7899 tablet_copy_client.cc:523] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:29.953156 7899 tablet_bootstrap.cc:492] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e: Bootstrap starting.
I20240503 02:14:29.956140 7891 log.cc:824] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:29.970986 7906 ts_tablet_manager.cc:902] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e: Initiating tablet copy from peer b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019)
I20240503 02:14:30.009299 7906 tablet_copy_client.cc:308] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.193:36019
I20240503 02:14:30.010031 6726 tablet_copy_service.cc:140] P b1ff4710a0a24719ae6a43233f261fb8: Received BeginTabletCopySession request for tablet 18bbda38b89544048c3b6cbae8d655db from peer 4fbc402737f64c5b9297ff0f7e22044e ({username='slave'} at 127.23.104.195:39709)
I20240503 02:14:30.010178 6726 tablet_copy_service.cc:161] P b1ff4710a0a24719ae6a43233f261fb8: Beginning new tablet copy session on tablet 18bbda38b89544048c3b6cbae8d655db from peer 4fbc402737f64c5b9297ff0f7e22044e at {username='slave'} at 127.23.104.195:39709: session id = 4fbc402737f64c5b9297ff0f7e22044e-18bbda38b89544048c3b6cbae8d655db
I20240503 02:14:30.012245 7891 tablet_bootstrap.cc:492] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:30.012782 7891 tablet_bootstrap.cc:492] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e: Bootstrap complete.
I20240503 02:14:30.013110 6726 tablet_copy_source_session.cc:205] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:30.013243 7891 ts_tablet_manager.cc:1367] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e: Time spent bootstrapping tablet: real 0.068s user 0.061s sys 0.005s
I20240503 02:14:30.014516 7906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18bbda38b89544048c3b6cbae8d655db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:30.020503 7891 raft_consensus.cc:348] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } }
I20240503 02:14:30.020987 7891 raft_consensus.cc:729] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4fbc402737f64c5b9297ff0f7e22044e, State: Initialized, Role: LEARNER
I20240503 02:14:30.021646 7891 consensus_queue.cc:260] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } }
I20240503 02:14:30.021975 7906 tablet_copy_client.cc:783] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:30.022331 7906 tablet_copy_client.cc:647] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:30.024528 7906 tablet_copy_client.cc:523] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:30.027910 7899 tablet_bootstrap.cc:492] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:30.029376 7906 tablet_bootstrap.cc:492] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e: Bootstrap starting.
I20240503 02:14:30.030781 7891 ts_tablet_manager.cc:1398] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e: Time spent starting tablet: real 0.017s user 0.010s sys 0.008s
I20240503 02:14:30.033043 6726 tablet_copy_service.cc:342] P b1ff4710a0a24719ae6a43233f261fb8: Request end of tablet copy session 4fbc402737f64c5b9297ff0f7e22044e-35c2b9629a3348bda52e6a64051d59da received from {username='slave'} at 127.23.104.195:39709
I20240503 02:14:30.033174 6726 tablet_copy_service.cc:433] P b1ff4710a0a24719ae6a43233f261fb8: ending tablet copy session 4fbc402737f64c5b9297ff0f7e22044e-35c2b9629a3348bda52e6a64051d59da on tablet 35c2b9629a3348bda52e6a64051d59da with peer 4fbc402737f64c5b9297ff0f7e22044e
I20240503 02:14:30.036172 7899 tablet_bootstrap.cc:492] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e: Bootstrap complete.
I20240503 02:14:30.036566 7899 ts_tablet_manager.cc:1367] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e: Time spent bootstrapping tablet: real 0.083s user 0.052s sys 0.006s
I20240503 02:14:30.037595 7899 raft_consensus.cc:348] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } }
I20240503 02:14:30.037870 7899 raft_consensus.cc:729] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4fbc402737f64c5b9297ff0f7e22044e, State: Initialized, Role: LEARNER
I20240503 02:14:30.038090 7899 consensus_queue.cc:260] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } }
I20240503 02:14:30.038627 7899 ts_tablet_manager.cc:1398] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:30.040895 6726 tablet_copy_service.cc:342] P b1ff4710a0a24719ae6a43233f261fb8: Request end of tablet copy session 4fbc402737f64c5b9297ff0f7e22044e-d0455e93fffe45b0a4253ff7d4bc7e14 received from {username='slave'} at 127.23.104.195:39709
I20240503 02:14:30.041041 6726 tablet_copy_service.cc:433] P b1ff4710a0a24719ae6a43233f261fb8: ending tablet copy session 4fbc402737f64c5b9297ff0f7e22044e-d0455e93fffe45b0a4253ff7d4bc7e14 on tablet d0455e93fffe45b0a4253ff7d4bc7e14 with peer 4fbc402737f64c5b9297ff0f7e22044e
I20240503 02:14:30.073813 7906 tablet_bootstrap.cc:492] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:30.074200 7906 tablet_bootstrap.cc:492] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e: Bootstrap complete.
I20240503 02:14:30.074446 7906 ts_tablet_manager.cc:1367] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e: Time spent bootstrapping tablet: real 0.045s user 0.039s sys 0.003s
I20240503 02:14:30.075105 7906 raft_consensus.cc:348] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } }
I20240503 02:14:30.075322 7906 raft_consensus.cc:729] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4fbc402737f64c5b9297ff0f7e22044e, State: Initialized, Role: LEARNER
I20240503 02:14:30.075500 7906 consensus_queue.cc:260] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } }
I20240503 02:14:30.076005 7906 ts_tablet_manager.cc:1398] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:14:30.076718 6726 tablet_copy_service.cc:342] P b1ff4710a0a24719ae6a43233f261fb8: Request end of tablet copy session 4fbc402737f64c5b9297ff0f7e22044e-18bbda38b89544048c3b6cbae8d655db received from {username='slave'} at 127.23.104.195:39709
I20240503 02:14:30.076861 6726 tablet_copy_service.cc:433] P b1ff4710a0a24719ae6a43233f261fb8: ending tablet copy session 4fbc402737f64c5b9297ff0f7e22044e-18bbda38b89544048c3b6cbae8d655db on tablet 18bbda38b89544048c3b6cbae8d655db with peer 4fbc402737f64c5b9297ff0f7e22044e
I20240503 02:14:30.273896 7451 raft_consensus.cc:1203] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:14:30.332051 7451 raft_consensus.cc:1203] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:14:30.346858 7582 raft_consensus.cc:1203] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:14:30.350499 7451 raft_consensus.cc:1203] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:14:30.368463 7582 raft_consensus.cc:1203] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:14:30.386617 7582 raft_consensus.cc:1203] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:14:30.752863 7944 raft_consensus.cc:1050] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8: attempting to promote NON_VOTER ee60d5902a7044f9b55e6702b4d8a29a to VOTER
I20240503 02:14:30.753702 7944 consensus_queue.cc:237] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } }
I20240503 02:14:30.755658 7102 raft_consensus.cc:1261] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Refusing update from remote peer b1ff4710a0a24719ae6a43233f261fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:30.755753 7451 raft_consensus.cc:1261] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 LEARNER]: Refusing update from remote peer b1ff4710a0a24719ae6a43233f261fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:30.756255 7943 consensus_queue.cc:1035] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:30.756551 7767 consensus_queue.cc:1035] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:30.758949 7767 raft_consensus.cc:2932] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.759622 7102 raft_consensus.cc:2932] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.760987 7451 raft_consensus.cc:2932] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.763872 6554 catalog_manager.cc:5484] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75 reported cstate change: config changed from index 4 to 5, ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.769574 6705 consensus_queue.cc:237] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } }
I20240503 02:14:30.771064 7451 raft_consensus.cc:1261] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 FOLLOWER]: Refusing update from remote peer b1ff4710a0a24719ae6a43233f261fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:30.771550 7767 consensus_queue.cc:1035] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:30.773640 7943 raft_consensus.cc:2932] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4334b0381be846cd801049e113b85b75 (127.23.104.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.774190 7451 raft_consensus.cc:2932] T 08cf83a491bd47cba73264964f6cf238 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4334b0381be846cd801049e113b85b75 (127.23.104.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.774871 7944 raft_consensus.cc:1050] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8: attempting to promote NON_VOTER 4fbc402737f64c5b9297ff0f7e22044e to VOTER
I20240503 02:14:30.775621 7944 consensus_queue.cc:237] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } }
I20240503 02:14:30.777521 6547 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08cf83a491bd47cba73264964f6cf238 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:14:30.778800 6555 catalog_manager.cc:5484] T 08cf83a491bd47cba73264964f6cf238 P b1ff4710a0a24719ae6a43233f261fb8 reported cstate change: config changed from index 5 to 6, VOTER 4334b0381be846cd801049e113b85b75 (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:14:30.780601 7102 raft_consensus.cc:1261] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Refusing update from remote peer b1ff4710a0a24719ae6a43233f261fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:30.780710 7582 raft_consensus.cc:1261] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e [term 1 LEARNER]: Refusing update from remote peer b1ff4710a0a24719ae6a43233f261fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:30.781340 7767 consensus_queue.cc:1035] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:30.781713 7944 consensus_queue.cc:1035] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:30.783123 7082 tablet_service.cc:1514] Processing DeleteTablet for tablet 08cf83a491bd47cba73264964f6cf238 with delete_type TABLET_DATA_TOMBSTONED (TS 4334b0381be846cd801049e113b85b75 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46186
I20240503 02:14:30.783975 8017 tablet_replica.cc:330] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75: stopping tablet replica
I20240503 02:14:30.784541 8017 raft_consensus.cc:2229] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:30.785185 8017 raft_consensus.cc:2258] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:30.787267 8017 ts_tablet_manager.cc:1875] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:30.788250 7767 raft_consensus.cc:2932] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:30.789059 7582 raft_consensus.cc:2932] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:30.794417 8017 ts_tablet_manager.cc:1888] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:14:30.794603 8017 log.cc:1196] T 08cf83a491bd47cba73264964f6cf238 P 4334b0381be846cd801049e113b85b75: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/08cf83a491bd47cba73264964f6cf238
I20240503 02:14:30.794519 6555 catalog_manager.cc:5484] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 reported cstate change: config changed from index 4 to 5, 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:14:30.795548 6547 catalog_manager.cc:4830] TS 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849): tablet 08cf83a491bd47cba73264964f6cf238 (table rebalance_test_table_0 [id=b3827a0d78b743c4bb95ec1161954529]) successfully deleted
I20240503 02:14:30.797509 7780 raft_consensus.cc:1050] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75: attempting to promote NON_VOTER ee60d5902a7044f9b55e6702b4d8a29a to VOTER
I20240503 02:14:30.797312 7102 raft_consensus.cc:2932] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:30.798491 7780 consensus_queue.cc:237] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } }
I20240503 02:14:30.800997 7451 raft_consensus.cc:1261] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 LEARNER]: Refusing update from remote peer 4334b0381be846cd801049e113b85b75: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:30.801399 6705 raft_consensus.cc:1261] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Refusing update from remote peer 4334b0381be846cd801049e113b85b75: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:30.801573 7970 consensus_queue.cc:1035] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:30.801890 7971 consensus_queue.cc:1035] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:30.814294 7970 raft_consensus.cc:2932] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.815245 6705 raft_consensus.cc:2932] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.815407 7451 raft_consensus.cc:2932] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.820237 6558 catalog_manager.cc:5484] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 reported cstate change: config changed from index 4 to 5, ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4334b0381be846cd801049e113b85b75" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:14:30.824937 7102 consensus_queue.cc:237] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } }
I20240503 02:14:30.827108 7451 raft_consensus.cc:1261] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 FOLLOWER]: Refusing update from remote peer 4334b0381be846cd801049e113b85b75: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:30.827633 7970 consensus_queue.cc:1035] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:30.829555 7780 raft_consensus.cc:2932] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.830200 7451 raft_consensus.cc:2932] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.832142 6705 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35c2b9629a3348bda52e6a64051d59da"
dest_uuid: "b1ff4710a0a24719ae6a43233f261fb8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60184
I20240503 02:14:30.832334 6705 raft_consensus.cc:595] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:14:30.833030 6547 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3bd8b03234ab4d6cb280c42bfc4e3f38 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:14:30.834615 6555 catalog_manager.cc:5484] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P 4334b0381be846cd801049e113b85b75 reported cstate change: config changed from index 5 to 6, VOTER b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "4334b0381be846cd801049e113b85b75" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:14:30.838411 6685 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bd8b03234ab4d6cb280c42bfc4e3f38 with delete_type TABLET_DATA_TOMBSTONED (TS b1ff4710a0a24719ae6a43233f261fb8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59888
I20240503 02:14:30.848266 8034 tablet_replica.cc:330] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8: stopping tablet replica
I20240503 02:14:30.848610 8034 raft_consensus.cc:2229] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:30.848934 8034 raft_consensus.cc:2258] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:30.850576 8034 ts_tablet_manager.cc:1875] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:30.856993 8034 ts_tablet_manager.cc:1888] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:14:30.857159 8034 log.cc:1196] T 3bd8b03234ab4d6cb280c42bfc4e3f38 P b1ff4710a0a24719ae6a43233f261fb8: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/3bd8b03234ab4d6cb280c42bfc4e3f38
I20240503 02:14:30.857992 6547 catalog_manager.cc:4830] TS b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019): tablet 3bd8b03234ab4d6cb280c42bfc4e3f38 (table rebalance_test_table_2 [id=c07dd68b46ac4aaf865ab28b1d7bf701]) successfully deleted
I20240503 02:14:30.858760 7780 raft_consensus.cc:1050] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75: attempting to promote NON_VOTER ee60d5902a7044f9b55e6702b4d8a29a to VOTER
I20240503 02:14:30.859535 7780 consensus_queue.cc:237] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } }
I20240503 02:14:30.861876 6705 raft_consensus.cc:1261] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Refusing update from remote peer 4334b0381be846cd801049e113b85b75: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:30.862433 7451 raft_consensus.cc:1261] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 LEARNER]: Refusing update from remote peer 4334b0381be846cd801049e113b85b75: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:30.862450 7971 consensus_queue.cc:1035] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:30.862982 7971 consensus_queue.cc:1035] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:30.865526 7780 raft_consensus.cc:2932] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.866839 7451 raft_consensus.cc:2932] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.867237 6705 raft_consensus.cc:2932] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.870482 6555 catalog_manager.cc:5484] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 reported cstate change: config changed from index 4 to 5, ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4334b0381be846cd801049e113b85b75" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:14:30.875327 7102 consensus_queue.cc:237] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } }
I20240503 02:14:30.877183 7451 raft_consensus.cc:1261] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 FOLLOWER]: Refusing update from remote peer 4334b0381be846cd801049e113b85b75: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:30.877636 7971 consensus_queue.cc:1035] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:30.879595 7780 raft_consensus.cc:2932] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.880537 7451 raft_consensus.cc:2932] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.881918 8016 raft_consensus.cc:1050] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8: attempting to promote NON_VOTER 4fbc402737f64c5b9297ff0f7e22044e to VOTER
I20240503 02:14:30.882462 8016 consensus_queue.cc:237] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } }
I20240503 02:14:30.883612 6547 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3bfc771e15b44977be9a9ea9c528b977 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:14:30.885133 6555 catalog_manager.cc:5484] T 3bfc771e15b44977be9a9ea9c528b977 P ee60d5902a7044f9b55e6702b4d8a29a reported cstate change: config changed from index 5 to 6, VOTER b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "4334b0381be846cd801049e113b85b75" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "ee60d5902a7044f9b55e6702b4d8a29a" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 33215 } attrs { promote: false } } }
I20240503 02:14:30.885353 7582 raft_consensus.cc:1261] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e [term 1 LEARNER]: Refusing update from remote peer b1ff4710a0a24719ae6a43233f261fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:30.885569 7102 raft_consensus.cc:1261] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Refusing update from remote peer b1ff4710a0a24719ae6a43233f261fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:30.885866 7767 consensus_queue.cc:1035] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:30.886752 7767 consensus_queue.cc:1035] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:30.889477 6685 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bfc771e15b44977be9a9ea9c528b977 with delete_type TABLET_DATA_TOMBSTONED (TS b1ff4710a0a24719ae6a43233f261fb8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59888
I20240503 02:14:30.890480 8034 tablet_replica.cc:330] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8: stopping tablet replica
I20240503 02:14:30.891021 8034 raft_consensus.cc:2229] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:30.891990 7944 raft_consensus.cc:2932] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:30.892692 7581 raft_consensus.cc:2932] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:30.895987 8034 raft_consensus.cc:2258] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:30.896649 6554 catalog_manager.cc:5484] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e reported cstate change: config changed from index 4 to 5, 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:30.897881 8034 ts_tablet_manager.cc:1875] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:30.902190 6705 consensus_queue.cc:237] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } }
I20240503 02:14:30.903178 7101 raft_consensus.cc:2932] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:30.905140 8034 ts_tablet_manager.cc:1888] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:14:30.905247 8034 log.cc:1196] T 3bfc771e15b44977be9a9ea9c528b977 P b1ff4710a0a24719ae6a43233f261fb8: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/3bfc771e15b44977be9a9ea9c528b977
I20240503 02:14:30.905880 6547 catalog_manager.cc:4830] TS b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019): tablet 3bfc771e15b44977be9a9ea9c528b977 (table rebalance_test_table_1 [id=5c3442b3ed234a8190878f87d52fec63]) successfully deleted
I20240503 02:14:30.908643 7582 raft_consensus.cc:1261] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e [term 1 FOLLOWER]: Refusing update from remote peer b1ff4710a0a24719ae6a43233f261fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:30.909559 7944 consensus_queue.cc:1035] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:30.911636 8016 raft_consensus.cc:2932] T d0455e93fffe45b0a4253ff7d4bc7e14 P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4334b0381be846cd801049e113b85b75 (127.23.104.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:30.912371 7582 raft_consensus.cc:2932] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4334b0381be846cd801049e113b85b75 (127.23.104.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:30.915073 6547 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d0455e93fffe45b0a4253ff7d4bc7e14 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:14:30.917241 6555 catalog_manager.cc:5484] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4fbc402737f64c5b9297ff0f7e22044e reported cstate change: config changed from index 5 to 6, VOTER 4334b0381be846cd801049e113b85b75 (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:30.922794 7082 tablet_service.cc:1514] Processing DeleteTablet for tablet d0455e93fffe45b0a4253ff7d4bc7e14 with delete_type TABLET_DATA_TOMBSTONED (TS 4334b0381be846cd801049e113b85b75 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46186
I20240503 02:14:30.923154 8017 tablet_replica.cc:330] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75: stopping tablet replica
I20240503 02:14:30.923334 8017 raft_consensus.cc:2229] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:30.923635 8017 raft_consensus.cc:2258] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:30.924721 8017 ts_tablet_manager.cc:1875] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:30.931505 8017 ts_tablet_manager.cc:1888] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:14:30.931633 8017 log.cc:1196] T d0455e93fffe45b0a4253ff7d4bc7e14 P 4334b0381be846cd801049e113b85b75: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/d0455e93fffe45b0a4253ff7d4bc7e14
I20240503 02:14:30.932359 6547 catalog_manager.cc:4830] TS 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849): tablet d0455e93fffe45b0a4253ff7d4bc7e14 (table rebalance_test_table_2 [id=c07dd68b46ac4aaf865ab28b1d7bf701]) successfully deleted
I20240503 02:14:30.960103 6705 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35c2b9629a3348bda52e6a64051d59da"
dest_uuid: "b1ff4710a0a24719ae6a43233f261fb8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60224
I20240503 02:14:30.960290 6705 raft_consensus.cc:595] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:14:30.966308 7767 raft_consensus.cc:1050] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8: attempting to promote NON_VOTER 4fbc402737f64c5b9297ff0f7e22044e to VOTER
I20240503 02:14:30.967059 7767 consensus_queue.cc:237] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } }
I20240503 02:14:30.969527 7101 raft_consensus.cc:1261] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Refusing update from remote peer b1ff4710a0a24719ae6a43233f261fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:30.970171 8016 consensus_queue.cc:1035] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:30.970636 7582 raft_consensus.cc:1261] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e [term 1 LEARNER]: Refusing update from remote peer b1ff4710a0a24719ae6a43233f261fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:30.971172 8016 consensus_queue.cc:1035] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:30.974226 7944 raft_consensus.cc:2932] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:30.974768 7582 raft_consensus.cc:2932] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:30.978883 6555 catalog_manager.cc:5484] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e reported cstate change: config changed from index 4 to 5, 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:30.983614 6705 consensus_queue.cc:237] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } }
I20240503 02:14:30.984993 7582 raft_consensus.cc:1261] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e [term 1 FOLLOWER]: Refusing update from remote peer b1ff4710a0a24719ae6a43233f261fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:30.985426 7767 consensus_queue.cc:1035] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:30.987234 8016 raft_consensus.cc:2932] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4334b0381be846cd801049e113b85b75 (127.23.104.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:30.987754 7582 raft_consensus.cc:2932] T 18bbda38b89544048c3b6cbae8d655db P 4fbc402737f64c5b9297ff0f7e22044e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4334b0381be846cd801049e113b85b75 (127.23.104.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:30.989702 6547 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 18bbda38b89544048c3b6cbae8d655db with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:14:30.991199 6558 catalog_manager.cc:5484] T 18bbda38b89544048c3b6cbae8d655db P b1ff4710a0a24719ae6a43233f261fb8 reported cstate change: config changed from index 5 to 6, VOTER 4334b0381be846cd801049e113b85b75 (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "b1ff4710a0a24719ae6a43233f261fb8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:14:30.993983 7082 tablet_service.cc:1514] Processing DeleteTablet for tablet 18bbda38b89544048c3b6cbae8d655db with delete_type TABLET_DATA_TOMBSTONED (TS 4334b0381be846cd801049e113b85b75 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46186
I20240503 02:14:30.994273 8017 tablet_replica.cc:330] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75: stopping tablet replica
I20240503 02:14:30.994474 8017 raft_consensus.cc:2229] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:30.994668 8017 pending_rounds.cc:62] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75: Trying to abort 1 pending ops.
I20240503 02:14:30.994758 8017 pending_rounds.cc:68] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023421321079160832 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "18bbda38b89544048c3b6cbae8d655db" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } attrs { replace: true } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } } }
I20240503 02:14:30.995024 8017 raft_consensus.cc:2894] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 02:14:30.995106 8017 raft_consensus.cc:2866] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:14:30.995189 8017 raft_consensus.cc:2258] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:30.996031 8017 ts_tablet_manager.cc:1875] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:31.000070 8017 ts_tablet_manager.cc:1888] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:14:31.000170 8017 log.cc:1196] T 18bbda38b89544048c3b6cbae8d655db P 4334b0381be846cd801049e113b85b75: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/18bbda38b89544048c3b6cbae8d655db
I20240503 02:14:31.000735 6547 catalog_manager.cc:4830] TS 4334b0381be846cd801049e113b85b75 (127.23.104.196:32849): tablet 18bbda38b89544048c3b6cbae8d655db (table rebalance_test_table_1 [id=5c3442b3ed234a8190878f87d52fec63]) successfully deleted
I20240503 02:14:31.187741 6705 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35c2b9629a3348bda52e6a64051d59da"
dest_uuid: "b1ff4710a0a24719ae6a43233f261fb8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60256
I20240503 02:14:31.187984 6705 raft_consensus.cc:595] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:14:31.211588 6705 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35c2b9629a3348bda52e6a64051d59da"
dest_uuid: "b1ff4710a0a24719ae6a43233f261fb8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60276
I20240503 02:14:31.211769 6705 raft_consensus.cc:595] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:14:31.348340 8016 raft_consensus.cc:979] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8: : Instructing follower 4334b0381be846cd801049e113b85b75 to start an election
I20240503 02:14:31.348508 8016 raft_consensus.cc:1067] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Signalling peer 4334b0381be846cd801049e113b85b75 to start an election
I20240503 02:14:31.349298 7101 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35c2b9629a3348bda52e6a64051d59da"
dest_uuid: "4334b0381be846cd801049e113b85b75"
from {username='slave'} at 127.23.104.193:55797
I20240503 02:14:31.349534 7101 raft_consensus.cc:482] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:14:31.349640 7101 raft_consensus.cc:3037] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:31.351619 7101 raft_consensus.cc:504] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } }
I20240503 02:14:31.352440 7101 leader_election.cc:288] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [CANDIDATE]: Term 2 election: Requested vote from peers b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019), 4fbc402737f64c5b9297ff0f7e22044e (127.23.104.195:32971)
I20240503 02:14:31.352846 6705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c2b9629a3348bda52e6a64051d59da" candidate_uuid: "4334b0381be846cd801049e113b85b75" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b1ff4710a0a24719ae6a43233f261fb8"
I20240503 02:14:31.353055 6705 raft_consensus.cc:3032] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:14:31.353164 6705 raft_consensus.cc:729] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b1ff4710a0a24719ae6a43233f261fb8, State: Running, Role: LEADER
I20240503 02:14:31.353454 6705 consensus_queue.cc:260] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } }
I20240503 02:14:31.353880 6705 raft_consensus.cc:3037] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:31.355839 6705 raft_consensus.cc:2454] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4334b0381be846cd801049e113b85b75 in term 2.
I20240503 02:14:31.356273 7034 leader_election.cc:302] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4334b0381be846cd801049e113b85b75, b1ff4710a0a24719ae6a43233f261fb8; no voters:
I20240503 02:14:31.356573 7780 raft_consensus.cc:2781] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:31.356757 7780 raft_consensus.cc:686] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 2 LEADER]: Becoming Leader. State: Replica: 4334b0381be846cd801049e113b85b75, State: Running, Role: LEADER
I20240503 02:14:31.357049 7780 consensus_queue.cc:237] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } }
I20240503 02:14:31.359079 7582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c2b9629a3348bda52e6a64051d59da" candidate_uuid: "4334b0381be846cd801049e113b85b75" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4fbc402737f64c5b9297ff0f7e22044e"
I20240503 02:14:31.359364 7582 raft_consensus.cc:3037] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:31.359530 6555 catalog_manager.cc:5484] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 reported cstate change: term changed from 1 to 2, leader changed from b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193) to 4334b0381be846cd801049e113b85b75 (127.23.104.196). New cstate: current_term: 2 leader_uuid: "4334b0381be846cd801049e113b85b75" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:31.361675 7582 raft_consensus.cc:2454] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4334b0381be846cd801049e113b85b75 in term 2.
I20240503 02:14:31.800467 7582 raft_consensus.cc:1261] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e [term 2 FOLLOWER]: Refusing update from remote peer 4334b0381be846cd801049e113b85b75: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:14:31.801021 7780 consensus_queue.cc:1035] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:31.804665 6705 raft_consensus.cc:1261] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 2 FOLLOWER]: Refusing update from remote peer 4334b0381be846cd801049e113b85b75: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:14:31.805367 7780 consensus_queue.cc:1035] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ff4710a0a24719ae6a43233f261fb8" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 36019 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:31.807516 7101 consensus_queue.cc:237] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } }
I20240503 02:14:31.808800 7582 raft_consensus.cc:1261] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e [term 2 FOLLOWER]: Refusing update from remote peer 4334b0381be846cd801049e113b85b75: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:14:31.809238 7780 consensus_queue.cc:1035] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:14:31.810995 7971 raft_consensus.cc:2932] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:31.811556 7582 raft_consensus.cc:2932] T 35c2b9629a3348bda52e6a64051d59da P 4fbc402737f64c5b9297ff0f7e22044e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } } }
I20240503 02:14:31.814512 6547 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 35c2b9629a3348bda52e6a64051d59da with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:14:31.815747 6555 catalog_manager.cc:5484] T 35c2b9629a3348bda52e6a64051d59da P 4334b0381be846cd801049e113b85b75 reported cstate change: config changed from index 5 to 7, VOTER b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193) evicted. New cstate: current_term: 2 leader_uuid: "4334b0381be846cd801049e113b85b75" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4334b0381be846cd801049e113b85b75" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 32849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fbc402737f64c5b9297ff0f7e22044e" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 32971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:14:31.818784 6685 tablet_service.cc:1514] Processing DeleteTablet for tablet 35c2b9629a3348bda52e6a64051d59da with delete_type TABLET_DATA_TOMBSTONED (TS b1ff4710a0a24719ae6a43233f261fb8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59888
I20240503 02:14:31.819278 8127 tablet_replica.cc:330] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8: stopping tablet replica
I20240503 02:14:31.819609 8127 raft_consensus.cc:2229] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:31.819849 8127 raft_consensus.cc:2258] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:31.820681 8127 ts_tablet_manager.cc:1875] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:31.824718 8127 ts_tablet_manager.cc:1888] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:14:31.824823 8127 log.cc:1196] T 35c2b9629a3348bda52e6a64051d59da P b1ff4710a0a24719ae6a43233f261fb8: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/35c2b9629a3348bda52e6a64051d59da
I20240503 02:14:31.825430 6547 catalog_manager.cc:4830] TS b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019): tablet 35c2b9629a3348bda52e6a64051d59da (table rebalance_test_table_0 [id=b3827a0d78b743c4bb95ec1161954529]) successfully deleted
I20240503 02:14:31.936445 7082 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:31.941125 6685 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:31.946484 7431 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:31.957765 7562 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:32.140919 7082 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:32.147157 7562 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:32.155267 7431 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:32.164505 6685 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:32.294730 7082 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:32.295430 7562 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:32.306032 6685 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:32.317085 7431 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:32.481475 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 6623
I20240503 02:14:32.498292 6745 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:32.624681 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 6623
W20240503 02:14:32.647863 7517 connection.cc:535] server connection from 127.23.104.193:59717 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:32.647863 7384 connection.cc:535] server connection from 127.23.104.193:55727 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:14:32.648568 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 7367
I20240503 02:14:32.654373 7490 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:32.756958 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 7367
W20240503 02:14:32.776976 7037 connection.cc:535] client connection to 127.23.104.194:33215 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:14:32.777108 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 7500
W20240503 02:14:32.777205 7037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:32.779641 7037 consensus_peers.cc:485] T 3bfc771e15b44977be9a9ea9c528b977 P 4334b0381be846cd801049e113b85b75 -> Peer ee60d5902a7044f9b55e6702b4d8a29a (127.23.104.194:33215): Couldn't send request to peer ee60d5902a7044f9b55e6702b4d8a29a. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.194:33215: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:32.783309 7621 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:14:32.803933 7034 consensus_peers.cc:485] T 02a869584fc946ad9447da0b5a2203b3 P 4334b0381be846cd801049e113b85b75 -> Peer b1ff4710a0a24719ae6a43233f261fb8 (127.23.104.193:36019): Couldn't send request to peer b1ff4710a0a24719ae6a43233f261fb8. Status: Network error: Client connection negotiation failed: client connection to 127.23.104.193:36019: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:32.887387 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 7500
W20240503 02:14:32.914731 6547 connection.cc:535] server connection from 127.23.104.195:51605 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:14:32.914912 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 7018
I20240503 02:14:32.922770 7142 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:33.044651 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 7018
I20240503 02:14:33.066747 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 6532
I20240503 02:14:33.067764 6594 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:33.195432 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 6532
2024-05-03T02:14:33Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1 (17695 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx6_rf7_t7_l_3_4_3
I20240503 02:14:33.246618 23971 test_util.cc:268] Using random seed: 1204828035
I20240503 02:14:33.248512 23971 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:14:33.248584 23971 ts_itest-base.cc:113] --------------
I20240503 02:14:33.248647 23971 ts_itest-base.cc:114] 10 tablet servers
I20240503 02:14:33.248693 23971 ts_itest-base.cc:115] 7 replicas per TS
I20240503 02:14:33.248764 23971 ts_itest-base.cc:116] --------------
2024-05-03T02:14:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:14:33Z Disabled control of system clock
I20240503 02:14:33.293071 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.104.254:36549
--webserver_interface=127.23.104.254
--webserver_port=0
--builtin_ntp_servers=127.23.104.212:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.104.254:36549
--location_mapping_cmd=/tmp/dist-test-taskTHQSCq/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 02:14:33.603546 8251 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 02:14:33.603904 8251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:33.603988 8251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:33.604058 8251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:33.610998 8251 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:33.611104 8251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:14:33.611173 8251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:33.611219 8251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:14:33.611277 8251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:14:33.620538 8251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskTHQSCq/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--ipki_ca_key_size=768
--master_addresses=127.23.104.254:36549
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.104.254:36549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.23.104.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:33.621757 8251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:33.623456 8251 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:33.633726 8259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:33.634281 8256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:33.634779 8257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:33.635329 8251 server_base.cc:1034] running on GCE node
I20240503 02:14:33.636018 8251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:33.637250 8251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:33.638455 8251 hybrid_clock.cc:648] HybridClock initialized: now 1714702473638401 us; error 37 us; skew 500 ppm
I20240503 02:14:33.638723 8251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:33.640820 8251 webserver.cc:460] Webserver started at http://127.23.104.254:40751/ using document root <none> and password file <none>
I20240503 02:14:33.641498 8251 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:33.641625 8251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:33.641899 8251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:33.644275 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a57a84b2a4e748779b71eead9c92b06b"
format_stamp: "Formatted at 2024-05-03 02:14:33 on dist-test-slave-38ht"
I20240503 02:14:33.644982 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a57a84b2a4e748779b71eead9c92b06b"
format_stamp: "Formatted at 2024-05-03 02:14:33 on dist-test-slave-38ht"
I20240503 02:14:33.649453 8251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:14:33.652619 8265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:33.653688 8251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240503 02:14:33.653837 8251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a57a84b2a4e748779b71eead9c92b06b"
format_stamp: "Formatted at 2024-05-03 02:14:33 on dist-test-slave-38ht"
I20240503 02:14:33.654026 8251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:33.666220 8251 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:33.666877 8251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:33.667176 8251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:33.683992 8251 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.254:36549
I20240503 02:14:33.684019 8317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.254:36549 every 8 connection(s)
I20240503 02:14:33.685160 8251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:14:33.687808 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 8251
I20240503 02:14:33.688055 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:14:33.688949 8318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:33.700408 8318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b: Bootstrap starting.
I20240503 02:14:33.703447 8318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:33.704537 8318 log.cc:824] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:33.706770 8318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b: No bootstrap required, opened a new log
I20240503 02:14:33.712811 8318 raft_consensus.cc:348] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57a84b2a4e748779b71eead9c92b06b" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 36549 } }
I20240503 02:14:33.713145 8318 raft_consensus.cc:374] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:33.713210 8318 raft_consensus.cc:729] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57a84b2a4e748779b71eead9c92b06b, State: Initialized, Role: FOLLOWER
I20240503 02:14:33.713737 8318 consensus_queue.cc:260] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57a84b2a4e748779b71eead9c92b06b" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 36549 } }
I20240503 02:14:33.713929 8318 raft_consensus.cc:388] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:33.714049 8318 raft_consensus.cc:482] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:33.714186 8318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:33.716490 8318 raft_consensus.cc:504] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57a84b2a4e748779b71eead9c92b06b" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 36549 } }
I20240503 02:14:33.716912 8318 leader_election.cc:302] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a57a84b2a4e748779b71eead9c92b06b; no voters:
I20240503 02:14:33.717429 8318 leader_election.cc:288] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:14:33.717527 8323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:33.718137 8323 raft_consensus.cc:686] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [term 1 LEADER]: Becoming Leader. State: Replica: a57a84b2a4e748779b71eead9c92b06b, State: Running, Role: LEADER
I20240503 02:14:33.718662 8323 consensus_queue.cc:237] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57a84b2a4e748779b71eead9c92b06b" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 36549 } }
I20240503 02:14:33.719130 8318 sys_catalog.cc:564] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:14:33.721391 8325 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a57a84b2a4e748779b71eead9c92b06b. Latest consensus state: current_term: 1 leader_uuid: "a57a84b2a4e748779b71eead9c92b06b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57a84b2a4e748779b71eead9c92b06b" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 36549 } } }
I20240503 02:14:33.721720 8325 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [sys.catalog]: This master's current role is: LEADER
I20240503 02:14:33.727640 8324 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a57a84b2a4e748779b71eead9c92b06b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57a84b2a4e748779b71eead9c92b06b" member_type: VOTER last_known_addr { host: "127.23.104.254" port: 36549 } } }
I20240503 02:14:33.728049 8324 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b [sys.catalog]: This master's current role is: LEADER
I20240503 02:14:33.729379 8329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:14:33.736393 8329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:14:33.744029 8329 catalog_manager.cc:1347] Generated new cluster ID: a3e763795dd04a41accb5c6d8dcd4dac
I20240503 02:14:33.744130 8329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:14:33.759176 8329 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:14:33.760284 8329 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:14:33.769382 8329 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a57a84b2a4e748779b71eead9c92b06b: Generated new TSK 0
I20240503 02:14:33.770022 8329 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:14:33.782967 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.193:0
--local_ip_for_outbound_sockets=127.23.104.193
--webserver_interface=127.23.104.193
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--builtin_ntp_servers=127.23.104.212:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:14:34.104470 8342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:34.104730 8342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:34.104863 8342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:34.112038 8342 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:34.112213 8342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:34.112419 8342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.193
I20240503 02:14:34.120792 8342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.23.104.193
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.193
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:34.121937 8342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:34.123555 8342 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:34.134680 8348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:34.135210 8347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:34.135730 8350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:34.136333 8342 server_base.cc:1034] running on GCE node
I20240503 02:14:34.136837 8342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:34.137635 8342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:34.138855 8342 hybrid_clock.cc:648] HybridClock initialized: now 1714702474138822 us; error 24 us; skew 500 ppm
I20240503 02:14:34.139147 8342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:34.141263 8342 webserver.cc:460] Webserver started at http://127.23.104.193:40211/ using document root <none> and password file <none>
I20240503 02:14:34.141924 8342 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:34.142037 8342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:34.142323 8342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:34.146490 8342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
format_stamp: "Formatted at 2024-05-03 02:14:34 on dist-test-slave-38ht"
I20240503 02:14:34.147215 8342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
format_stamp: "Formatted at 2024-05-03 02:14:34 on dist-test-slave-38ht"
I20240503 02:14:34.151903 8342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20240503 02:14:34.154789 8356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:34.156026 8342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:14:34.156174 8342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
format_stamp: "Formatted at 2024-05-03 02:14:34 on dist-test-slave-38ht"
I20240503 02:14:34.156358 8342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:34.203377 8342 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:34.204206 8342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:34.204461 8342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:34.205334 8342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:34.206830 8342 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:34.206930 8342 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:34.207044 8342 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:34.207115 8342 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:34.237260 8342 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.193:38635
I20240503 02:14:34.237318 8469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.193:38635 every 8 connection(s)
I20240503 02:14:34.238490 8342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:14:34.241252 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 8342
I20240503 02:14:34.241540 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:14:34.244810 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.194:0
--local_ip_for_outbound_sockets=127.23.104.194
--webserver_interface=127.23.104.194
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--builtin_ntp_servers=127.23.104.212:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:34.257227 8470 heartbeater.cc:347] Connected to a master server at 127.23.104.254:36549
I20240503 02:14:34.257557 8470 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:34.258442 8470 heartbeater.cc:511] Master 127.23.104.254:36549 requested a full tablet report, sending...
I20240503 02:14:34.302534 8282 ts_manager.cc:194] Registered new tserver with Master: 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635)
I20240503 02:14:34.304325 8282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.193:37649
W20240503 02:14:34.566298 8474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:34.566586 8474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:34.566723 8474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:34.574041 8474 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:34.574187 8474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:34.574345 8474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.194
I20240503 02:14:34.582923 8474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.23.104.194
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.194
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:34.584179 8474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:34.585801 8474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:34.597154 8480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:34.597730 8481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:34.598219 8483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:34.598814 8474 server_base.cc:1034] running on GCE node
I20240503 02:14:34.599300 8474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:34.600162 8474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:34.601343 8474 hybrid_clock.cc:648] HybridClock initialized: now 1714702474601294 us; error 43 us; skew 500 ppm
I20240503 02:14:34.601637 8474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:34.603717 8474 webserver.cc:460] Webserver started at http://127.23.104.194:37999/ using document root <none> and password file <none>
I20240503 02:14:34.604396 8474 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:34.604512 8474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:34.604812 8474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:34.607200 8474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2a1893ab239f4163a06bdbf8ce579431"
format_stamp: "Formatted at 2024-05-03 02:14:34 on dist-test-slave-38ht"
I20240503 02:14:34.607944 8474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2a1893ab239f4163a06bdbf8ce579431"
format_stamp: "Formatted at 2024-05-03 02:14:34 on dist-test-slave-38ht"
I20240503 02:14:34.612632 8474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:14:34.615545 8489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:34.616704 8474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:14:34.616847 8474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2a1893ab239f4163a06bdbf8ce579431"
format_stamp: "Formatted at 2024-05-03 02:14:34 on dist-test-slave-38ht"
I20240503 02:14:34.617017 8474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:34.635236 8474 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:34.636005 8474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:34.636245 8474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:34.637099 8474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:34.638447 8474 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:34.638540 8474 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:34.638650 8474 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:34.638721 8474 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:34.669505 8474 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.194:45923
I20240503 02:14:34.669553 8602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.194:45923 every 8 connection(s)
I20240503 02:14:34.670734 8474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:14:34.677481 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 8474
I20240503 02:14:34.677652 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:14:34.680822 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.195:0
--local_ip_for_outbound_sockets=127.23.104.195
--webserver_interface=127.23.104.195
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--builtin_ntp_servers=127.23.104.212:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:34.682725 8603 heartbeater.cc:347] Connected to a master server at 127.23.104.254:36549
I20240503 02:14:34.682979 8603 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:34.683952 8603 heartbeater.cc:511] Master 127.23.104.254:36549 requested a full tablet report, sending...
I20240503 02:14:34.736591 8282 ts_manager.cc:194] Registered new tserver with Master: 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923)
I20240503 02:14:34.737970 8282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.194:50677
W20240503 02:14:35.009397 8607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:35.009671 8607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:35.009774 8607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:35.017089 8607 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:35.017199 8607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:35.017311 8607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.195
I20240503 02:14:35.026175 8607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.23.104.195
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.195
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:35.027323 8607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:35.028931 8607 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:35.040637 8615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:35.041247 8613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:35.041746 8612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:35.042318 8607 server_base.cc:1034] running on GCE node
I20240503 02:14:35.042814 8607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:35.043578 8607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:35.044757 8607 hybrid_clock.cc:648] HybridClock initialized: now 1714702475044702 us; error 50 us; skew 500 ppm
I20240503 02:14:35.045035 8607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:35.047106 8607 webserver.cc:460] Webserver started at http://127.23.104.195:33333/ using document root <none> and password file <none>
I20240503 02:14:35.047773 8607 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:35.047909 8607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:35.048182 8607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:35.050516 8607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
format_stamp: "Formatted at 2024-05-03 02:14:35 on dist-test-slave-38ht"
I20240503 02:14:35.051229 8607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
format_stamp: "Formatted at 2024-05-03 02:14:35 on dist-test-slave-38ht"
I20240503 02:14:35.055999 8607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20240503 02:14:35.059072 8621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:35.060336 8607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:14:35.060475 8607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
format_stamp: "Formatted at 2024-05-03 02:14:35 on dist-test-slave-38ht"
I20240503 02:14:35.060675 8607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:35.078514 8607 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:35.079244 8607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:35.079520 8607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:35.080415 8607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:35.081777 8607 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:35.081874 8607 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:35.081982 8607 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:35.082053 8607 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:35.112248 8607 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.195:45495
I20240503 02:14:35.112309 8734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.195:45495 every 8 connection(s)
I20240503 02:14:35.113516 8607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:14:35.121726 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 8607
I20240503 02:14:35.121943 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:14:35.125239 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.196:0
--local_ip_for_outbound_sockets=127.23.104.196
--webserver_interface=127.23.104.196
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--builtin_ntp_servers=127.23.104.212:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:35.125784 8735 heartbeater.cc:347] Connected to a master server at 127.23.104.254:36549
I20240503 02:14:35.126085 8735 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:35.127068 8735 heartbeater.cc:511] Master 127.23.104.254:36549 requested a full tablet report, sending...
I20240503 02:14:35.171936 8282 ts_manager.cc:194] Registered new tserver with Master: 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:35.172900 8282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.195:33913
I20240503 02:14:35.307816 8470 heartbeater.cc:503] Master 127.23.104.254:36549 was elected leader, sending a full tablet report...
W20240503 02:14:35.445036 8739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:35.445291 8739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:35.445376 8739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:35.452349 8739 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:35.452452 8739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:35.452570 8739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.196
I20240503 02:14:35.460867 8739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.23.104.196
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.196
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:35.462051 8739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:35.463636 8739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:35.474824 8744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:35.475306 8745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:35.475849 8747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:35.476452 8739 server_base.cc:1034] running on GCE node
I20240503 02:14:35.476972 8739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:35.477815 8739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:35.479024 8739 hybrid_clock.cc:648] HybridClock initialized: now 1714702475478987 us; error 35 us; skew 500 ppm
I20240503 02:14:35.479321 8739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:35.481431 8739 webserver.cc:460] Webserver started at http://127.23.104.196:34993/ using document root <none> and password file <none>
I20240503 02:14:35.482111 8739 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:35.482226 8739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:35.482519 8739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:35.484985 8739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "79e41f25066e4e5abbd71e48d545c6bf"
format_stamp: "Formatted at 2024-05-03 02:14:35 on dist-test-slave-38ht"
I20240503 02:14:35.485689 8739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "79e41f25066e4e5abbd71e48d545c6bf"
format_stamp: "Formatted at 2024-05-03 02:14:35 on dist-test-slave-38ht"
I20240503 02:14:35.490412 8739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:14:35.493410 8753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:35.494545 8739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:14:35.494683 8739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "79e41f25066e4e5abbd71e48d545c6bf"
format_stamp: "Formatted at 2024-05-03 02:14:35 on dist-test-slave-38ht"
I20240503 02:14:35.494850 8739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:35.522332 8739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:35.523087 8739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:35.523304 8739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:35.524189 8739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:35.525518 8739 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:35.525601 8739 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:35.525682 8739 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:35.525733 8739 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:35.552332 8739 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.196:34747
I20240503 02:14:35.552395 8865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.196:34747 every 8 connection(s)
I20240503 02:14:35.553495 8739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:14:35.554492 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 8739
I20240503 02:14:35.554692 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:14:35.558193 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.197:0
--local_ip_for_outbound_sockets=127.23.104.197
--webserver_interface=127.23.104.197
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--builtin_ntp_servers=127.23.104.212:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:35.567553 8866 heartbeater.cc:347] Connected to a master server at 127.23.104.254:36549
I20240503 02:14:35.567834 8866 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:35.568569 8866 heartbeater.cc:511] Master 127.23.104.254:36549 requested a full tablet report, sending...
I20240503 02:14:35.611323 8282 ts_manager.cc:194] Registered new tserver with Master: 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747)
I20240503 02:14:35.612653 8282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.196:41919
I20240503 02:14:35.740743 8603 heartbeater.cc:503] Master 127.23.104.254:36549 was elected leader, sending a full tablet report...
W20240503 02:14:35.881254 8869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:35.881557 8869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:35.881644 8869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:35.888656 8869 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:35.888762 8869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:35.888871 8869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.197
I20240503 02:14:35.897172 8869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.23.104.197
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.197
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:35.898373 8869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:35.900174 8869 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:35.911845 8876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:35.912281 8875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:35.912817 8878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:35.913528 8869 server_base.cc:1034] running on GCE node
I20240503 02:14:35.914044 8869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:35.914868 8869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:35.916050 8869 hybrid_clock.cc:648] HybridClock initialized: now 1714702475916002 us; error 39 us; skew 500 ppm
I20240503 02:14:35.916317 8869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:35.918262 8869 webserver.cc:460] Webserver started at http://127.23.104.197:46617/ using document root <none> and password file <none>
I20240503 02:14:35.918946 8869 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:35.919054 8869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:35.919385 8869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:35.922009 8869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "99be4ff1114d4aaabcc537604fce54b3"
format_stamp: "Formatted at 2024-05-03 02:14:35 on dist-test-slave-38ht"
I20240503 02:14:35.922744 8869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "99be4ff1114d4aaabcc537604fce54b3"
format_stamp: "Formatted at 2024-05-03 02:14:35 on dist-test-slave-38ht"
I20240503 02:14:35.927487 8869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20240503 02:14:35.930537 8884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:35.931725 8869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:35.931888 8869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "99be4ff1114d4aaabcc537604fce54b3"
format_stamp: "Formatted at 2024-05-03 02:14:35 on dist-test-slave-38ht"
I20240503 02:14:35.932061 8869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:35.952018 8869 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:35.952754 8869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:35.952989 8869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:35.953831 8869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:35.955139 8869 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:35.955235 8869 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:35.955343 8869 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:35.955410 8869 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:35.986045 8869 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.197:45403
I20240503 02:14:35.986115 8996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.197:45403 every 8 connection(s)
I20240503 02:14:35.987378 8869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:14:35.989781 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 8869
I20240503 02:14:35.989989 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:14:35.993638 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.198:0
--local_ip_for_outbound_sockets=127.23.104.198
--webserver_interface=127.23.104.198
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--builtin_ntp_servers=127.23.104.212:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:35.999862 8997 heartbeater.cc:347] Connected to a master server at 127.23.104.254:36549
I20240503 02:14:36.000177 8997 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:36.000909 8997 heartbeater.cc:511] Master 127.23.104.254:36549 requested a full tablet report, sending...
I20240503 02:14:36.058382 8282 ts_manager.cc:194] Registered new tserver with Master: 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403)
I20240503 02:14:36.059351 8282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.197:41631
I20240503 02:14:36.175412 8735 heartbeater.cc:503] Master 127.23.104.254:36549 was elected leader, sending a full tablet report...
W20240503 02:14:36.323760 9000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:36.324043 9000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:36.324164 9000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:36.330647 9000 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:36.330765 9000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:36.330893 9000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.198
I20240503 02:14:36.338975 9000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.23.104.198
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.198
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:36.340164 9000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:36.341725 9000 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:36.352476 9006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:36.353020 9007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:36.353482 9009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:36.354131 9000 server_base.cc:1034] running on GCE node
I20240503 02:14:36.354665 9000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:36.355618 9000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:36.356817 9000 hybrid_clock.cc:648] HybridClock initialized: now 1714702476356772 us; error 48 us; skew 500 ppm
I20240503 02:14:36.357115 9000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:36.359293 9000 webserver.cc:460] Webserver started at http://127.23.104.198:39439/ using document root <none> and password file <none>
I20240503 02:14:36.360077 9000 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:36.360216 9000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:36.360514 9000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:36.363013 9000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c399efcbba924863ac89d156dc07ac7d"
format_stamp: "Formatted at 2024-05-03 02:14:36 on dist-test-slave-38ht"
I20240503 02:14:36.363734 9000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c399efcbba924863ac89d156dc07ac7d"
format_stamp: "Formatted at 2024-05-03 02:14:36 on dist-test-slave-38ht"
I20240503 02:14:36.368542 9000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20240503 02:14:36.371526 9015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:36.372753 9000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:14:36.372900 9000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c399efcbba924863ac89d156dc07ac7d"
format_stamp: "Formatted at 2024-05-03 02:14:36 on dist-test-slave-38ht"
I20240503 02:14:36.373081 9000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:36.399377 9000 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:36.400206 9000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:36.400455 9000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:36.401325 9000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:36.402702 9000 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:36.402810 9000 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:36.402922 9000 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:36.402999 9000 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:36.434065 9000 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.198:41237
I20240503 02:14:36.434130 9127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.198:41237 every 8 connection(s)
I20240503 02:14:36.435518 9000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:14:36.437019 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 9000
I20240503 02:14:36.437182 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:14:36.448802 9128 heartbeater.cc:347] Connected to a master server at 127.23.104.254:36549
I20240503 02:14:36.449082 9128 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:36.449872 9128 heartbeater.cc:511] Master 127.23.104.254:36549 requested a full tablet report, sending...
I20240503 02:14:36.476428 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.199:0
--local_ip_for_outbound_sockets=127.23.104.199
--webserver_interface=127.23.104.199
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--builtin_ntp_servers=127.23.104.212:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:36.493491 8282 ts_manager.cc:194] Registered new tserver with Master: c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:14:36.494390 8282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.198:59495
I20240503 02:14:36.615530 8866 heartbeater.cc:503] Master 127.23.104.254:36549 was elected leader, sending a full tablet report...
W20240503 02:14:36.800681 9132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:36.800937 9132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:36.801074 9132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:36.808189 9132 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:36.808310 9132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:36.808451 9132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.199
I20240503 02:14:36.816787 9132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.23.104.199
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.199
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:36.818018 9132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:36.819770 9132 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:36.832015 9137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:36.832818 9138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:36.833398 9132 server_base.cc:1034] running on GCE node
W20240503 02:14:36.833851 9140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:36.834410 9132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:36.835275 9132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:36.836477 9132 hybrid_clock.cc:648] HybridClock initialized: now 1714702476836404 us; error 70 us; skew 500 ppm
I20240503 02:14:36.836742 9132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:36.839057 9132 webserver.cc:460] Webserver started at http://127.23.104.199:40339/ using document root <none> and password file <none>
I20240503 02:14:36.839743 9132 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:36.839887 9132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:36.840164 9132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:36.842542 9132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
format_stamp: "Formatted at 2024-05-03 02:14:36 on dist-test-slave-38ht"
I20240503 02:14:36.843227 9132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
format_stamp: "Formatted at 2024-05-03 02:14:36 on dist-test-slave-38ht"
I20240503 02:14:36.848007 9132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20240503 02:14:36.850991 9146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:36.852272 9132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:14:36.852402 9132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
format_stamp: "Formatted at 2024-05-03 02:14:36 on dist-test-slave-38ht"
I20240503 02:14:36.852574 9132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:36.887554 9132 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:36.888329 9132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:36.888556 9132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:36.889433 9132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:36.890784 9132 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:36.890879 9132 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:36.890952 9132 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:36.891002 9132 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:36.921890 9132 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.199:32809
I20240503 02:14:36.921998 9258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.199:32809 every 8 connection(s)
I20240503 02:14:36.923158 9132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:14:36.924757 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 9132
I20240503 02:14:36.924993 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:14:36.928421 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.200:0
--local_ip_for_outbound_sockets=127.23.104.200
--webserver_interface=127.23.104.200
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--builtin_ntp_servers=127.23.104.212:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:36.936460 9259 heartbeater.cc:347] Connected to a master server at 127.23.104.254:36549
I20240503 02:14:36.936764 9259 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:36.937561 9259 heartbeater.cc:511] Master 127.23.104.254:36549 requested a full tablet report, sending...
I20240503 02:14:37.019968 8282 ts_manager.cc:194] Registered new tserver with Master: cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:37.021303 8282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.199:38571
I20240503 02:14:37.062448 8997 heartbeater.cc:503] Master 127.23.104.254:36549 was elected leader, sending a full tablet report...
W20240503 02:14:37.260354 9262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:37.260627 9262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:37.260758 9262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:37.267874 9262 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:37.267990 9262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:37.268144 9262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.200
I20240503 02:14:37.276690 9262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.23.104.200
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.200
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:37.278014 9262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:37.279687 9262 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:37.291257 9268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:37.291829 9271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:37.292282 9269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:37.292903 9262 server_base.cc:1034] running on GCE node
I20240503 02:14:37.293443 9262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:37.294274 9262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:37.295469 9262 hybrid_clock.cc:648] HybridClock initialized: now 1714702477295429 us; error 50 us; skew 500 ppm
I20240503 02:14:37.295776 9262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:37.297876 9262 webserver.cc:460] Webserver started at http://127.23.104.200:44021/ using document root <none> and password file <none>
I20240503 02:14:37.298564 9262 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:37.298672 9262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:37.299000 9262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:37.301513 9262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df"
format_stamp: "Formatted at 2024-05-03 02:14:37 on dist-test-slave-38ht"
I20240503 02:14:37.302218 9262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df"
format_stamp: "Formatted at 2024-05-03 02:14:37 on dist-test-slave-38ht"
I20240503 02:14:37.306905 9262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240503 02:14:37.310062 9277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:37.311223 9262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:37.311372 9262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df"
format_stamp: "Formatted at 2024-05-03 02:14:37 on dist-test-slave-38ht"
I20240503 02:14:37.311556 9262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:37.323937 9262 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:37.324653 9262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:37.324903 9262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:37.325723 9262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:37.327039 9262 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:37.327136 9262 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:37.327246 9262 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:37.327316 9262 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:37.357501 9262 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.200:45027
I20240503 02:14:37.357556 9390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.200:45027 every 8 connection(s)
I20240503 02:14:37.358716 9262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:14:37.359145 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 9262
I20240503 02:14:37.359314 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:14:37.363044 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.201:0
--local_ip_for_outbound_sockets=127.23.104.201
--webserver_interface=127.23.104.201
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--builtin_ntp_servers=127.23.104.212:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:37.372637 9391 heartbeater.cc:347] Connected to a master server at 127.23.104.254:36549
I20240503 02:14:37.372926 9391 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:37.373672 9391 heartbeater.cc:511] Master 127.23.104.254:36549 requested a full tablet report, sending...
I20240503 02:14:37.414937 8282 ts_manager.cc:194] Registered new tserver with Master: e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200:45027)
I20240503 02:14:37.416316 8282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.200:40067
I20240503 02:14:37.497221 9128 heartbeater.cc:503] Master 127.23.104.254:36549 was elected leader, sending a full tablet report...
W20240503 02:14:37.695139 9394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:37.695492 9394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:37.695628 9394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:37.703037 9394 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:37.703150 9394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:37.703305 9394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.201
I20240503 02:14:37.711755 9394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.201:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.23.104.201
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.201
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:37.712903 9394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:37.714561 9394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:37.726981 9401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:37.727483 9403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:37.728055 9400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:37.729231 9394 server_base.cc:1034] running on GCE node
I20240503 02:14:37.729892 9394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:37.730775 9394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:37.731977 9394 hybrid_clock.cc:648] HybridClock initialized: now 1714702477731922 us; error 52 us; skew 500 ppm
I20240503 02:14:37.732262 9394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:37.734254 9394 webserver.cc:460] Webserver started at http://127.23.104.201:34709/ using document root <none> and password file <none>
I20240503 02:14:37.734927 9394 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:37.735036 9394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:37.735337 9394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:37.737780 9394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "4792082063bd4d58bc222c9c95b16335"
format_stamp: "Formatted at 2024-05-03 02:14:37 on dist-test-slave-38ht"
I20240503 02:14:37.738468 9394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "4792082063bd4d58bc222c9c95b16335"
format_stamp: "Formatted at 2024-05-03 02:14:37 on dist-test-slave-38ht"
I20240503 02:14:37.743202 9394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240503 02:14:37.746501 9409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:37.747674 9394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:14:37.747861 9394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "4792082063bd4d58bc222c9c95b16335"
format_stamp: "Formatted at 2024-05-03 02:14:37 on dist-test-slave-38ht"
I20240503 02:14:37.748071 9394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:37.765908 9394 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:37.766623 9394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:37.766861 9394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:37.767709 9394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:37.769061 9394 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:37.769155 9394 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:37.769266 9394 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:37.769336 9394 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:37.800029 9394 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.201:46877
I20240503 02:14:37.800072 9521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.201:46877 every 8 connection(s)
I20240503 02:14:37.801213 9394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240503 02:14:37.804914 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 9394
I20240503 02:14:37.805101 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240503 02:14:37.808267 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.202:0
--local_ip_for_outbound_sockets=127.23.104.202
--webserver_interface=127.23.104.202
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--builtin_ntp_servers=127.23.104.212:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:37.814342 9522 heartbeater.cc:347] Connected to a master server at 127.23.104.254:36549
I20240503 02:14:37.814640 9522 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:37.815573 9522 heartbeater.cc:511] Master 127.23.104.254:36549 requested a full tablet report, sending...
I20240503 02:14:37.860608 8282 ts_manager.cc:194] Registered new tserver with Master: 4792082063bd4d58bc222c9c95b16335 (127.23.104.201:46877)
I20240503 02:14:37.861928 8282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.201:39251
I20240503 02:14:38.024016 9259 heartbeater.cc:503] Master 127.23.104.254:36549 was elected leader, sending a full tablet report...
W20240503 02:14:38.141765 9525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:38.142050 9525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:38.142196 9525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:38.149351 9525 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:38.149469 9525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:38.149613 9525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.202
I20240503 02:14:38.158555 9525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.202:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.23.104.202
--webserver_port=0
--tserver_master_addrs=127.23.104.254:36549
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.202
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:38.159868 9525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:38.161525 9525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:38.172820 9531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:38.173341 9532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:38.173848 9534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:38.174424 9525 server_base.cc:1034] running on GCE node
I20240503 02:14:38.174949 9525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:38.175890 9525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:38.177112 9525 hybrid_clock.cc:648] HybridClock initialized: now 1714702478177036 us; error 63 us; skew 500 ppm
I20240503 02:14:38.177434 9525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:38.183598 9525 webserver.cc:460] Webserver started at http://127.23.104.202:39537/ using document root <none> and password file <none>
I20240503 02:14:38.184284 9525 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:38.184389 9525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:38.184669 9525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:38.187032 9525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/data/instance:
uuid: "dabeea74472746538132a520710248ea"
format_stamp: "Formatted at 2024-05-03 02:14:38 on dist-test-slave-38ht"
I20240503 02:14:38.187736 9525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/wal/instance:
uuid: "dabeea74472746538132a520710248ea"
format_stamp: "Formatted at 2024-05-03 02:14:38 on dist-test-slave-38ht"
I20240503 02:14:38.192540 9525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 02:14:38.195684 9540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:38.196852 9525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 02:14:38.196985 9525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "dabeea74472746538132a520710248ea"
format_stamp: "Formatted at 2024-05-03 02:14:38 on dist-test-slave-38ht"
I20240503 02:14:38.197165 9525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:38.224377 9525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:38.225098 9525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:38.225358 9525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:38.226227 9525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:38.227629 9525 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:38.227708 9525 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:38.227939 9525 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:38.228019 9525 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:38.259311 9525 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.202:37759
I20240503 02:14:38.259373 9652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.202:37759 every 8 connection(s)
I20240503 02:14:38.260759 9525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240503 02:14:38.262333 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 9525
I20240503 02:14:38.262526 23971 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/wal/instance
I20240503 02:14:38.272416 9653 heartbeater.cc:347] Connected to a master server at 127.23.104.254:36549
I20240503 02:14:38.272656 9653 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:38.273353 9653 heartbeater.cc:511] Master 127.23.104.254:36549 requested a full tablet report, sending...
I20240503 02:14:38.317763 8282 ts_manager.cc:194] Registered new tserver with Master: dabeea74472746538132a520710248ea (127.23.104.202:37759)
I20240503 02:14:38.318768 8282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.202:37195
I20240503 02:14:38.320271 23971 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240503 02:14:38.335393 23971 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240503 02:14:38.335520 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 9262
I20240503 02:14:38.342849 9386 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:38.418691 9391 heartbeater.cc:503] Master 127.23.104.254:36549 was elected leader, sending a full tablet report...
I20240503 02:14:38.418869 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 9262
I20240503 02:14:38.435276 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 9394
I20240503 02:14:38.443209 9517 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:38.521530 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 9394
I20240503 02:14:38.537482 23971 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu pid 9525
I20240503 02:14:38.545758 9648 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:38.625380 23971 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu with pid 9525
I20240503 02:14:42.393119 8282 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42220:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:14:42.399643 8282 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 02:14:42.438155 8537 tablet_service.cc:1467] Processing CreateTablet for tablet fc3f83e9a30047989c4281e049f0a68c (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.439581 8537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3f83e9a30047989c4281e049f0a68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.439833 8536 tablet_service.cc:1467] Processing CreateTablet for tablet 879167ff15db41a7b4ef21fa10c44b27 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.440383 8536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879167ff15db41a7b4ef21fa10c44b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.437808 8538 tablet_service.cc:1467] Processing CreateTablet for tablet 9b41db0e8b114c99ab1d8734ec8b008f (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.441593 8538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b41db0e8b114c99ab1d8734ec8b008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.449857 8405 tablet_service.cc:1467] Processing CreateTablet for tablet 9b41db0e8b114c99ab1d8734ec8b008f (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.450712 8404 tablet_service.cc:1467] Processing CreateTablet for tablet fc3f83e9a30047989c4281e049f0a68c (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.449829 8403 tablet_service.cc:1467] Processing CreateTablet for tablet 879167ff15db41a7b4ef21fa10c44b27 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.451617 8404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3f83e9a30047989c4281e049f0a68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.452090 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879167ff15db41a7b4ef21fa10c44b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.445940 8932 tablet_service.cc:1467] Processing CreateTablet for tablet 9b41db0e8b114c99ab1d8734ec8b008f (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.454809 8932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b41db0e8b114c99ab1d8734ec8b008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.456251 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b41db0e8b114c99ab1d8734ec8b008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.458014 8801 tablet_service.cc:1467] Processing CreateTablet for tablet 9b41db0e8b114c99ab1d8734ec8b008f (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.458050 8800 tablet_service.cc:1467] Processing CreateTablet for tablet fc3f83e9a30047989c4281e049f0a68c (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.459347 8801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b41db0e8b114c99ab1d8734ec8b008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.460014 8799 tablet_service.cc:1467] Processing CreateTablet for tablet 879167ff15db41a7b4ef21fa10c44b27 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.460822 8799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879167ff15db41a7b4ef21fa10c44b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.461043 8800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3f83e9a30047989c4281e049f0a68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.459491 9194 tablet_service.cc:1467] Processing CreateTablet for tablet 9b41db0e8b114c99ab1d8734ec8b008f (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.462194 9194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b41db0e8b114c99ab1d8734ec8b008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.465175 8931 tablet_service.cc:1467] Processing CreateTablet for tablet fc3f83e9a30047989c4281e049f0a68c (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.465718 8931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3f83e9a30047989c4281e049f0a68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.474833 8670 tablet_service.cc:1467] Processing CreateTablet for tablet 9b41db0e8b114c99ab1d8734ec8b008f (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.477236 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 879167ff15db41a7b4ef21fa10c44b27 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.478734 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879167ff15db41a7b4ef21fa10c44b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.480125 9063 tablet_service.cc:1467] Processing CreateTablet for tablet 9b41db0e8b114c99ab1d8734ec8b008f (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.480603 9063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b41db0e8b114c99ab1d8734ec8b008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.480770 8930 tablet_service.cc:1467] Processing CreateTablet for tablet 879167ff15db41a7b4ef21fa10c44b27 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.481379 8930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879167ff15db41a7b4ef21fa10c44b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.483358 9062 tablet_service.cc:1467] Processing CreateTablet for tablet fc3f83e9a30047989c4281e049f0a68c (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.483776 9062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3f83e9a30047989c4281e049f0a68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.485369 8670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b41db0e8b114c99ab1d8734ec8b008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.492482 9193 tablet_service.cc:1467] Processing CreateTablet for tablet fc3f83e9a30047989c4281e049f0a68c (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.493374 9193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3f83e9a30047989c4281e049f0a68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.494632 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 879167ff15db41a7b4ef21fa10c44b27 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.495136 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879167ff15db41a7b4ef21fa10c44b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.479921 8668 tablet_service.cc:1467] Processing CreateTablet for tablet 879167ff15db41a7b4ef21fa10c44b27 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.476186 8669 tablet_service.cc:1467] Processing CreateTablet for tablet fc3f83e9a30047989c4281e049f0a68c (DEFAULT_TABLE table=rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:42.497818 8669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3f83e9a30047989c4281e049f0a68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.503181 8668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879167ff15db41a7b4ef21fa10c44b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:42.556008 9687 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:42.564488 9688 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:42.564177 9690 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:42.568845 9690 tablet_bootstrap.cc:654] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.569339 9691 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:42.569911 9690 log.cc:824] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:42.571734 9687 tablet_bootstrap.cc:654] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.572614 9692 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:42.572700 9687 log.cc:824] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:42.577643 9687 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:42.577811 9689 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:42.577999 9686 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:42.578220 9687 ts_tablet_manager.cc:1367] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.003s
I20240503 02:14:42.579185 9690 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:42.579505 9690 ts_tablet_manager.cc:1367] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240503 02:14:42.581334 9686 tablet_bootstrap.cc:654] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.581334 9689 tablet_bootstrap.cc:654] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.582588 9689 log.cc:824] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:42.582594 9686 log.cc:824] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:42.585388 9691 tablet_bootstrap.cc:654] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.586670 9691 log.cc:824] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:42.588140 9687 raft_consensus.cc:348] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.588138 9690 raft_consensus.cc:348] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.588651 9690 raft_consensus.cc:374] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.588657 9688 tablet_bootstrap.cc:654] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.588761 9690 raft_consensus.cc:729] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.588840 9687 raft_consensus.cc:374] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.589053 9687 raft_consensus.cc:729] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.589628 9690 consensus_queue.cc:260] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.590119 9688 log.cc:824] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:42.589857 9687 consensus_queue.cc:260] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.591709 9692 tablet_bootstrap.cc:654] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.592959 9692 log.cc:824] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:42.594446 9686 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:42.594846 9686 ts_tablet_manager.cc:1367] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20240503 02:14:42.597915 9690 ts_tablet_manager.cc:1398] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.018s user 0.012s sys 0.007s
I20240503 02:14:42.598445 9690 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:42.598964 9691 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:42.599318 9691 ts_tablet_manager.cc:1367] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.004s
I20240503 02:14:42.600437 9690 tablet_bootstrap.cc:654] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.602198 9689 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:42.602586 9689 ts_tablet_manager.cc:1367] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20240503 02:14:42.603590 9699 raft_consensus.cc:482] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:42.603881 9699 raft_consensus.cc:504] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.606117 9688 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:42.606415 9688 ts_tablet_manager.cc:1367] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.002s
I20240503 02:14:42.615875 9688 raft_consensus.cc:348] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.616428 9688 raft_consensus.cc:374] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.616546 9688 raft_consensus.cc:729] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.617532 9688 consensus_queue.cc:260] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.620859 9699 leader_election.cc:288] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:14:42.621026 9688 ts_tablet_manager.cc:1398] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20240503 02:14:42.621436 9688 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:42.625603 9692 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:42.625910 9692 ts_tablet_manager.cc:1367] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.054s user 0.012s sys 0.007s
I20240503 02:14:42.628726 9690 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:42.628924 9690 ts_tablet_manager.cc:1367] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20240503 02:14:42.628767 9691 raft_consensus.cc:348] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.629315 9691 raft_consensus.cc:374] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.630043 9690 raft_consensus.cc:348] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.630324 9690 raft_consensus.cc:374] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.630427 9690 raft_consensus.cc:729] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.631017 9688 tablet_bootstrap.cc:654] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.630857 9690 consensus_queue.cc:260] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.631603 9690 ts_tablet_manager.cc:1398] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 02:14:42.631966 9690 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:42.633638 9690 tablet_bootstrap.cc:654] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.634518 9686 raft_consensus.cc:348] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.635039 9686 raft_consensus.cc:374] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.635138 9686 raft_consensus.cc:729] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.637337 9687 ts_tablet_manager.cc:1398] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.059s user 0.009s sys 0.008s
I20240503 02:14:42.638257 9692 raft_consensus.cc:348] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.638222 9689 raft_consensus.cc:348] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.638749 9692 raft_consensus.cc:374] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.638749 9689 raft_consensus.cc:374] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.638856 9689 raft_consensus.cc:729] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.638856 9692 raft_consensus.cc:729] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.629432 9691 raft_consensus.cc:729] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.639923 9687 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:42.640458 9691 consensus_queue.cc:260] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.641022 9692 consensus_queue.cc:260] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.641798 9686 consensus_queue.cc:260] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.642745 9689 consensus_queue.cc:260] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.643081 9688 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:42.643270 9688 ts_tablet_manager.cc:1367] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.005s
I20240503 02:14:42.644510 9688 raft_consensus.cc:348] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.644855 9688 raft_consensus.cc:374] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.644963 9688 raft_consensus.cc:729] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.645318 9688 consensus_queue.cc:260] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.646142 9688 ts_tablet_manager.cc:1398] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:14:42.646435 9688 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:42.646234 8952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
W20240503 02:14:42.647636 8756 leader_election.cc:341] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:42.648486 9688 tablet_bootstrap.cc:654] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.651567 9687 tablet_bootstrap.cc:654] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.651929 9692 ts_tablet_manager.cc:1398] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.026s user 0.012s sys 0.002s
I20240503 02:14:42.652110 9691 ts_tablet_manager.cc:1398] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.053s user 0.015s sys 0.003s
I20240503 02:14:42.652422 9692 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:42.654507 9692 tablet_bootstrap.cc:654] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.655951 9691 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:42.658983 9690 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:42.659178 9690 ts_tablet_manager.cc:1367] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240503 02:14:42.659262 9691 tablet_bootstrap.cc:654] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.660106 9690 raft_consensus.cc:348] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.660423 9690 raft_consensus.cc:374] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.660524 9690 raft_consensus.cc:729] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.661229 9690 consensus_queue.cc:260] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.661633 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:42.662041 9690 ts_tablet_manager.cc:1398] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:14:42.662079 9688 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:42.662194 8425 raft_consensus.cc:2454] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 0.
I20240503 02:14:42.662258 9688 ts_tablet_manager.cc:1367] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240503 02:14:42.663056 9688 raft_consensus.cc:348] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.663470 9688 raft_consensus.cc:374] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.663564 9688 raft_consensus.cc:729] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.663910 9698 raft_consensus.cc:482] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:42.663933 9688 consensus_queue.cc:260] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.664139 9698 raft_consensus.cc:504] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.665581 9713 raft_consensus.cc:482] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:42.665808 9713 raft_consensus.cc:504] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.667039 9698 leader_election.cc:288] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:42.668546 9688 ts_tablet_manager.cc:1398] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
W20240503 02:14:42.677954 9260 tablet.cc:2369] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:14:42.686262 9689 ts_tablet_manager.cc:1398] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.084s user 0.013s sys 0.006s
I20240503 02:14:42.686781 9689 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:42.689296 9686 ts_tablet_manager.cc:1398] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.094s user 0.017s sys 0.003s
I20240503 02:14:42.689788 9686 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
W20240503 02:14:42.691542 9129 tablet.cc:2369] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:14:42.697338 9686 tablet_bootstrap.cc:654] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.698951 9692 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:42.699163 9692 ts_tablet_manager.cc:1367] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20240503 02:14:42.704654 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:42.704773 9692 raft_consensus.cc:348] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.705062 9692 raft_consensus.cc:374] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.705175 9692 raft_consensus.cc:729] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.705535 9692 consensus_queue.cc:260] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.706259 9692 ts_tablet_manager.cc:1398] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:14:42.706566 9692 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
W20240503 02:14:42.707010 8757 leader_election.cc:341] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:42.708498 9692 tablet_bootstrap.cc:654] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.709573 9689 tablet_bootstrap.cc:654] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.715503 9713 leader_election.cc:288] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:14:42.716073 9691 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:42.716262 9691 ts_tablet_manager.cc:1367] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.001s
I20240503 02:14:42.717278 9691 raft_consensus.cc:348] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.717592 9691 raft_consensus.cc:374] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.717695 9691 raft_consensus.cc:729] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.726413 9686 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:42.728956 9691 consensus_queue.cc:260] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.729460 9710 raft_consensus.cc:482] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:42.729974 9710 raft_consensus.cc:504] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.733803 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:42.741714 9711 raft_consensus.cc:482] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:42.742084 9711 raft_consensus.cc:504] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
W20240503 02:14:42.743420 8998 tablet.cc:2369] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:14:42.746640 9711 leader_election.cc:288] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
W20240503 02:14:42.746706 8471 tablet.cc:2369] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:14:42.763357 9710 leader_election.cc:288] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:42.772886 9687 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:42.773170 9687 ts_tablet_manager.cc:1367] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.133s user 0.004s sys 0.004s
I20240503 02:14:42.774371 9687 raft_consensus.cc:348] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.774760 9687 raft_consensus.cc:374] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.774925 9687 raft_consensus.cc:729] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.775318 9687 consensus_queue.cc:260] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.777823 9687 ts_tablet_manager.cc:1398] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:14:42.778221 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:42.778720 9083 raft_consensus.cc:2454] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 0.
I20240503 02:14:42.783277 9692 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:42.783528 9692 ts_tablet_manager.cc:1367] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.077s user 0.021s sys 0.010s
I20240503 02:14:42.786584 9689 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:42.786772 9689 ts_tablet_manager.cc:1367] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.100s user 0.004s sys 0.000s
I20240503 02:14:42.787824 9689 raft_consensus.cc:348] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.788034 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:42.788123 9689 raft_consensus.cc:374] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.788218 9689 raft_consensus.cc:729] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.788262 8425 raft_consensus.cc:2454] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:42.788532 9689 consensus_queue.cc:260] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.789680 9689 ts_tablet_manager.cc:1398] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 02:14:42.789990 9689 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:42.790153 9687 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:42.726614 9686 ts_tablet_manager.cc:1367] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20240503 02:14:42.799943 9687 tablet_bootstrap.cc:654] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.800226 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:42.800384 9686 raft_consensus.cc:348] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.800664 9686 raft_consensus.cc:374] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.800717 8690 raft_consensus.cc:2454] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 0.
I20240503 02:14:42.800771 9686 raft_consensus.cc:729] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.801066 9686 consensus_queue.cc:260] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.801324 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:42.801667 8558 raft_consensus.cc:2454] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:42.802136 9691 ts_tablet_manager.cc:1398] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.086s user 0.001s sys 0.002s
I20240503 02:14:42.802242 8755 leader_election.cc:302] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, 846e89dd0b154d2b9bb7d4f315c9fc12, c399efcbba924863ac89d156dc07ac7d; no voters: 2a1893ab239f4163a06bdbf8ce579431, 99be4ff1114d4aaabcc537604fce54b3
I20240503 02:14:42.802445 9691 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:42.802685 9699 raft_consensus.cc:2781] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:42.802832 9699 raft_consensus.cc:482] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:42.802932 9699 raft_consensus.cc:3037] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.804194 9691 tablet_bootstrap.cc:654] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.806087 9699 raft_consensus.cc:504] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.806591 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:42.807051 8821 raft_consensus.cc:2454] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:42.807436 9699 leader_election.cc:288] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 election: Requested vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:14:42.807982 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:42.808220 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:42.808246 8558 raft_consensus.cc:3037] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.808483 8425 raft_consensus.cc:3037] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.810472 9687 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
W20240503 02:14:42.810786 8867 tablet.cc:2369] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:14:42.811575 8425 raft_consensus.cc:2454] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 1.
I20240503 02:14:42.811619 8558 raft_consensus.cc:2454] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 1.
I20240503 02:14:42.812076 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:42.812309 8690 raft_consensus.cc:3037] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.812592 9687 ts_tablet_manager.cc:1367] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
I20240503 02:14:42.813623 9687 raft_consensus.cc:348] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.814926 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:42.815109 8952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:42.815328 8424 raft_consensus.cc:2379] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in current term 1: Already voted for candidate 79e41f25066e4e5abbd71e48d545c6bf in this term.
I20240503 02:14:42.815397 8952 raft_consensus.cc:2454] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:42.816948 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:42.817168 9083 raft_consensus.cc:2454] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 0.
I20240503 02:14:42.815817 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:42.817701 8821 raft_consensus.cc:2379] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in current term 1: Already voted for candidate 79e41f25066e4e5abbd71e48d545c6bf in this term.
W20240503 02:14:42.816059 8757 leader_election.cc:341] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:42.818675 8690 raft_consensus.cc:2454] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 1.
I20240503 02:14:42.819202 8755 leader_election.cc:302] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, 846e89dd0b154d2b9bb7d4f315c9fc12; no voters:
I20240503 02:14:42.819944 9699 raft_consensus.cc:2781] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:42.820289 9699 raft_consensus.cc:686] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEADER]: Becoming Leader. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Running, Role: LEADER
I20240503 02:14:42.820145 9687 raft_consensus.cc:374] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.820783 9687 raft_consensus.cc:729] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.820899 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:42.821159 9083 raft_consensus.cc:3037] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.821038 9699 consensus_queue.cc:237] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.824421 9083 raft_consensus.cc:2454] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 1.
I20240503 02:14:42.824752 9019 leader_election.cc:302] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, c399efcbba924863ac89d156dc07ac7d; no voters:
I20240503 02:14:42.825263 9698 raft_consensus.cc:2781] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:42.825407 9698 raft_consensus.cc:482] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:42.825533 9698 raft_consensus.cc:3037] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.785534 9692 raft_consensus.cc:348] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.828393 9692 raft_consensus.cc:374] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.828490 9692 raft_consensus.cc:729] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.828624 9698 raft_consensus.cc:504] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.828791 9692 consensus_queue.cc:260] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.829972 9698 leader_election.cc:288] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 election: Requested vote from peers 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:42.830148 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:42.830374 8425 raft_consensus.cc:3037] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.831030 8952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:42.831143 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:42.831266 8952 raft_consensus.cc:3037] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.831389 8558 raft_consensus.cc:3037] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.821712 9687 consensus_queue.cc:260] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.833492 8425 raft_consensus.cc:2454] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:42.834362 8558 raft_consensus.cc:2454] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:42.834326 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:42.834435 8952 raft_consensus.cc:2454] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 1.
I20240503 02:14:42.835731 8951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:42.836125 8951 raft_consensus.cc:3037] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.837221 8952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:42.834533 8821 raft_consensus.cc:3037] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.837529 8952 raft_consensus.cc:2379] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in current term 1: Already voted for candidate 79e41f25066e4e5abbd71e48d545c6bf in this term.
I20240503 02:14:42.838605 9692 ts_tablet_manager.cc:1398] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.055s user 0.003s sys 0.000s
I20240503 02:14:42.839569 8951 raft_consensus.cc:2454] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:42.841614 8821 raft_consensus.cc:2454] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:42.843590 9689 tablet_bootstrap.cc:654] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.844115 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:42.844146 9017 leader_election.cc:302] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, c399efcbba924863ac89d156dc07ac7d; no voters:
I20240503 02:14:42.844467 8558 raft_consensus.cc:2379] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99be4ff1114d4aaabcc537604fce54b3 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
I20240503 02:14:42.844666 9698 raft_consensus.cc:2781] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:42.845515 8281 catalog_manager.cc:5484] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196). New cstate: current_term: 1 leader_uuid: "79e41f25066e4e5abbd71e48d545c6bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:42.849329 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:42.850766 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
W20240503 02:14:42.853369 9017 leader_election.cc:341] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:14:42.853731 9017 leader_election.cc:341] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:42.853698 8951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879167ff15db41a7b4ef21fa10c44b27" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:42.853946 8951 raft_consensus.cc:2454] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
I20240503 02:14:42.854126 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:42.854707 8558 raft_consensus.cc:2379] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in current term 1: Already voted for candidate 79e41f25066e4e5abbd71e48d545c6bf in this term.
I20240503 02:14:42.855391 9698 raft_consensus.cc:686] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Becoming Leader. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Running, Role: LEADER
I20240503 02:14:42.855612 9686 ts_tablet_manager.cc:1398] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:14:42.855661 8625 leader_election.cc:302] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 846e89dd0b154d2b9bb7d4f315c9fc12, c399efcbba924863ac89d156dc07ac7d; no voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3
I20240503 02:14:42.855967 9686 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:42.857932 9686 tablet_bootstrap.cc:654] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:42.858378 9698 consensus_queue.cc:237] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.862454 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879167ff15db41a7b4ef21fa10c44b27" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:42.867488 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:42.834439 9687 ts_tablet_manager.cc:1398] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:14:42.870790 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:42.871161 9214 raft_consensus.cc:2454] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
W20240503 02:14:42.871758 8736 tablet.cc:2369] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:14:42.875252 9691 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:42.875631 9691 ts_tablet_manager.cc:1367] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.000s
I20240503 02:14:42.876742 9691 raft_consensus.cc:348] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
W20240503 02:14:42.875483 9150 leader_election.cc:341] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:42.877431 9691 raft_consensus.cc:374] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.877558 9691 raft_consensus.cc:729] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.877910 9691 consensus_queue.cc:260] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:14:42.878949 9691 ts_tablet_manager.cc:1398] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:14:42.887121 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:42.887488 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:42.887769 8425 raft_consensus.cc:2379] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99be4ff1114d4aaabcc537604fce54b3 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
W20240503 02:14:42.891276 8886 leader_election.cc:341] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:42.892890 9686 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:42.893102 9686 ts_tablet_manager.cc:1367] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20240503 02:14:42.893945 9686 raft_consensus.cc:348] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.894277 9686 raft_consensus.cc:374] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.894361 9686 raft_consensus.cc:729] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.894587 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:42.894676 9686 consensus_queue.cc:260] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.894908 9214 raft_consensus.cc:2454] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:42.895673 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:42.895965 9214 raft_consensus.cc:3037] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.897768 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:42.898685 8886 leader_election.cc:302] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865; no voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 846e89dd0b154d2b9bb7d4f315c9fc12, c399efcbba924863ac89d156dc07ac7d
I20240503 02:14:42.899189 9711 raft_consensus.cc:2726] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:42.899866 9686 ts_tablet_manager.cc:1398] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:14:42.900492 9214 raft_consensus.cc:2454] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:42.901911 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:42.902197 8821 raft_consensus.cc:2379] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99be4ff1114d4aaabcc537604fce54b3 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
I20240503 02:14:42.901830 8280 catalog_manager.cc:5484] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c399efcbba924863ac89d156dc07ac7d (127.23.104.198). New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:42.902356 9713 raft_consensus.cc:2726] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:14:42.902717 8625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.104.194:45923, user_credentials={real_user=slave}} blocked reactor thread for 47104.6us
W20240503 02:14:42.903244 8625 leader_election.cc:341] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:42.905211 9689 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:42.905424 9689 ts_tablet_manager.cc:1367] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.115s user 0.002s sys 0.003s
I20240503 02:14:42.906275 9689 raft_consensus.cc:348] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.906611 9689 raft_consensus.cc:374] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:42.906687 9689 raft_consensus.cc:729] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:42.906968 9689 consensus_queue.cc:260] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:42.907608 9689 ts_tablet_manager.cc:1398] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 02:14:42.909919 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b41db0e8b114c99ab1d8734ec8b008f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:42.910193 9214 raft_consensus.cc:3037] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.911088 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879167ff15db41a7b4ef21fa10c44b27" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:42.911319 8425 raft_consensus.cc:2454] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
I20240503 02:14:42.912196 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879167ff15db41a7b4ef21fa10c44b27" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:42.912395 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879167ff15db41a7b4ef21fa10c44b27" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:42.912503 8821 raft_consensus.cc:2454] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
I20240503 02:14:42.912588 8690 raft_consensus.cc:2454] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
I20240503 02:14:42.912974 9214 raft_consensus.cc:2454] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 1.
I20240503 02:14:42.913373 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879167ff15db41a7b4ef21fa10c44b27" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:42.913666 9083 raft_consensus.cc:2454] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
I20240503 02:14:42.913942 9148 leader_election.cc:302] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865; no voters: 2a1893ab239f4163a06bdbf8ce579431
I20240503 02:14:42.914326 9710 raft_consensus.cc:2781] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:42.914459 9710 raft_consensus.cc:482] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:42.914566 9710 raft_consensus.cc:3037] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.916700 9710 raft_consensus.cc:504] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.917713 9710 leader_election.cc:288] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 election: Requested vote from peers 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:42.917879 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879167ff15db41a7b4ef21fa10c44b27" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:42.917919 8951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879167ff15db41a7b4ef21fa10c44b27" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:42.918140 8821 raft_consensus.cc:3037] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.918176 8951 raft_consensus.cc:3037] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.918390 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879167ff15db41a7b4ef21fa10c44b27" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:42.918579 8558 raft_consensus.cc:3037] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.920253 8951 raft_consensus.cc:2454] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:42.920874 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879167ff15db41a7b4ef21fa10c44b27" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:42.921123 8425 raft_consensus.cc:3037] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.921111 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879167ff15db41a7b4ef21fa10c44b27" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:42.921301 8690 raft_consensus.cc:3037] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.921345 8558 raft_consensus.cc:2454] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:42.921314 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879167ff15db41a7b4ef21fa10c44b27" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:42.921545 9083 raft_consensus.cc:3037] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:42.924376 8690 raft_consensus.cc:2454] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:42.924759 8821 raft_consensus.cc:2454] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:42.925276 8425 raft_consensus.cc:2454] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:42.925379 9083 raft_consensus.cc:2454] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:42.925819 9150 leader_election.cc:302] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 846e89dd0b154d2b9bb7d4f315c9fc12, 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:42.926285 9710 raft_consensus.cc:2781] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election won for term 1
W20240503 02:14:42.927268 8604 tablet.cc:2369] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:14:42.926909 9710 raft_consensus.cc:686] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Becoming Leader. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Running, Role: LEADER
I20240503 02:14:42.928478 9710 consensus_queue.cc:237] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:42.935379 8281 catalog_manager.cc:5484] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199). New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:43.013084 9713 raft_consensus.cc:482] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:43.013331 9713 raft_consensus.cc:504] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:43.014863 9713 leader_election.cc:288] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:43.015347 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:43.015643 8821 raft_consensus.cc:2379] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
I20240503 02:14:43.015594 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:43.015926 8425 raft_consensus.cc:2379] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
I20240503 02:14:43.016369 8951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:43.016654 8951 raft_consensus.cc:2379] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
I20240503 02:14:43.016803 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:43.017124 9214 raft_consensus.cc:2379] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
I20240503 02:14:43.017295 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:43.018190 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3f83e9a30047989c4281e049f0a68c" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:43.018437 8624 leader_election.cc:302] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 846e89dd0b154d2b9bb7d4f315c9fc12; no voters: 4c668b2e2d5c4edba46cbc5c36065c0c, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d, cb90f34dc37b4b16b2a5b5a4ee012865
I20240503 02:14:43.018563 8558 raft_consensus.cc:2379] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
I20240503 02:14:43.019058 9713 raft_consensus.cc:3037] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:43.021317 9713 raft_consensus.cc:2726] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:43.267205 9699 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.276943 9698 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.295670 9699 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.300588 9698 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.316397 9710 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.319167 9710 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.341318 9710 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.356200 9699 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.356448 9710 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.356858 9710 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.363759 9762 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.386368 9698 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.407166 9788 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.411286 9782 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.421783 9699 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.438410 9799 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:43.438815 9799 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:43.444655 9764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:43.445137 9764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:43.461136 9764 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:14:43.463125 9798 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 388.2 ms
time per row: 0.1941 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:14:44.945461 8280 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42220:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:14:44.946169 8280 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 02:14:44.954586 8801 tablet_service.cc:1467] Processing CreateTablet for tablet 99e42dca2db54de5a9417df20242e2c2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.954826 8931 tablet_service.cc:1467] Processing CreateTablet for tablet 99e42dca2db54de5a9417df20242e2c2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.955041 8801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e42dca2db54de5a9417df20242e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.955230 8931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e42dca2db54de5a9417df20242e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.955217 9194 tablet_service.cc:1467] Processing CreateTablet for tablet 99e42dca2db54de5a9417df20242e2c2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.955602 9194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e42dca2db54de5a9417df20242e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.957118 8537 tablet_service.cc:1467] Processing CreateTablet for tablet 99e42dca2db54de5a9417df20242e2c2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.957530 8537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e42dca2db54de5a9417df20242e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.960264 9063 tablet_service.cc:1467] Processing CreateTablet for tablet 99e42dca2db54de5a9417df20242e2c2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.960448 8670 tablet_service.cc:1467] Processing CreateTablet for tablet 99e42dca2db54de5a9417df20242e2c2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.960692 9063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e42dca2db54de5a9417df20242e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.960867 8670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e42dca2db54de5a9417df20242e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.964174 8405 tablet_service.cc:1467] Processing CreateTablet for tablet 99e42dca2db54de5a9417df20242e2c2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.964172 9061 tablet_service.cc:1467] Processing CreateTablet for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.964529 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.964553 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e42dca2db54de5a9417df20242e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.970041 8404 tablet_service.cc:1467] Processing CreateTablet for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.970378 8930 tablet_service.cc:1467] Processing CreateTablet for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.970553 8404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.970772 8930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.976341 8800 tablet_service.cc:1467] Processing CreateTablet for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.976745 8800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.976866 8669 tablet_service.cc:1467] Processing CreateTablet for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.977223 8669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.978785 8932 tablet_service.cc:1467] Processing CreateTablet for tablet b71357f9f8ef449189f485b3246869f1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.979249 8932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71357f9f8ef449189f485b3246869f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.981566 9193 tablet_service.cc:1467] Processing CreateTablet for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.981978 9193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.983371 8536 tablet_service.cc:1467] Processing CreateTablet for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.983765 8536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.994670 9877 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:44.995664 8800 tablet_service.cc:1467] Processing CreateTablet for tablet b71357f9f8ef449189f485b3246869f1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:44.996182 8800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71357f9f8ef449189f485b3246869f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:44.998730 9875 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:44.999620 9877 tablet_bootstrap.cc:654] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.001519 9875 tablet_bootstrap.cc:654] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.002368 9874 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:45.004573 9874 tablet_bootstrap.cc:654] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.006705 9062 tablet_service.cc:1467] Processing CreateTablet for tablet b71357f9f8ef449189f485b3246869f1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:45.007184 9062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71357f9f8ef449189f485b3246869f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:45.011322 8405 tablet_service.cc:1467] Processing CreateTablet for tablet b71357f9f8ef449189f485b3246869f1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:45.011780 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71357f9f8ef449189f485b3246869f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:45.016232 8537 tablet_service.cc:1467] Processing CreateTablet for tablet b71357f9f8ef449189f485b3246869f1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:45.016728 8537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71357f9f8ef449189f485b3246869f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:45.018014 9872 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:45.021064 9872 tablet_bootstrap.cc:654] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.024044 9194 tablet_service.cc:1467] Processing CreateTablet for tablet b71357f9f8ef449189f485b3246869f1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:45.024547 9194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71357f9f8ef449189f485b3246869f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:45.025367 9877 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:45.025368 9875 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:45.025506 9875 ts_tablet_manager.cc:1367] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.005s
I20240503 02:14:45.025615 9877 ts_tablet_manager.cc:1367] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.000s
I20240503 02:14:45.026338 9877 raft_consensus.cc:348] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.026589 9877 raft_consensus.cc:374] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.026667 9877 raft_consensus.cc:729] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.026914 9877 consensus_queue.cc:260] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.027318 8669 tablet_service.cc:1467] Processing CreateTablet for tablet b71357f9f8ef449189f485b3246869f1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:45.027640 9876 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:45.027732 8669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71357f9f8ef449189f485b3246869f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:45.028512 9875 raft_consensus.cc:348] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.030319 9876 tablet_bootstrap.cc:654] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.031164 9875 raft_consensus.cc:374] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.031688 9875 raft_consensus.cc:729] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.032603 9875 consensus_queue.cc:260] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.042091 9872 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:45.043632 9872 ts_tablet_manager.cc:1367] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.004s
I20240503 02:14:45.045076 9872 raft_consensus.cc:348] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.045737 9879 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:45.046309 9875 ts_tablet_manager.cc:1398] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.019s user 0.003s sys 0.001s
I20240503 02:14:45.047475 9875 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:45.050060 9872 raft_consensus.cc:374] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.050431 9872 raft_consensus.cc:729] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.051200 9872 consensus_queue.cc:260] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.053346 9872 ts_tablet_manager.cc:1398] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 02:14:45.055805 9882 raft_consensus.cc:482] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:45.056733 9875 tablet_bootstrap.cc:654] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.058574 9882 raft_consensus.cc:504] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.059201 9873 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:45.059705 9872 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:45.061712 9874 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:45.056130 9877 ts_tablet_manager.cc:1398] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.030s user 0.023s sys 0.004s
I20240503 02:14:45.061972 9874 ts_tablet_manager.cc:1367] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.005s
I20240503 02:14:45.062175 9877 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:45.062636 9699 raft_consensus.cc:482] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:45.062851 9699 raft_consensus.cc:504] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.063205 9879 tablet_bootstrap.cc:654] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.063773 9877 tablet_bootstrap.cc:654] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.064486 9699 leader_election.cc:288] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:45.065941 8557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:45.066162 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:45.066424 8950 raft_consensus.cc:2454] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 0.
I20240503 02:14:45.066951 9882 leader_election.cc:288] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:45.066890 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:45.067319 9872 tablet_bootstrap.cc:654] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.071251 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:45.071823 9873 tablet_bootstrap.cc:654] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.072057 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:45.072893 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
W20240503 02:14:45.073084 8757 leader_election.cc:341] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:14:45.074350 8755 leader_election.cc:341] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:14:45.074577 8755 leader_election.cc:341] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:14:45.076555 8757 leader_election.cc:341] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:45.076725 8757 leader_election.cc:302] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3; no voters: 2a1893ab239f4163a06bdbf8ce579431, 846e89dd0b154d2b9bb7d4f315c9fc12, c399efcbba924863ac89d156dc07ac7d, cb90f34dc37b4b16b2a5b5a4ee012865
I20240503 02:14:45.084075 9699 raft_consensus.cc:2726] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:45.088862 9875 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:45.089082 9875 ts_tablet_manager.cc:1367] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.001s
I20240503 02:14:45.089305 9874 raft_consensus.cc:348] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.089596 9874 raft_consensus.cc:374] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 02:14:45.089615 8755 leader_election.cc:341] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:45.089713 9874 raft_consensus.cc:729] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.089874 9876 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:45.089808 9875 raft_consensus.cc:348] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.090011 9876 ts_tablet_manager.cc:1367] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.062s user 0.019s sys 0.038s
I20240503 02:14:45.090099 9875 raft_consensus.cc:374] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.089937 9874 consensus_queue.cc:260] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.090183 9875 raft_consensus.cc:729] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.090433 9875 consensus_queue.cc:260] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.090754 9876 raft_consensus.cc:348] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.091020 9875 ts_tablet_manager.cc:1398] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:14:45.091022 9876 raft_consensus.cc:374] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.091105 9876 raft_consensus.cc:729] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.091342 9875 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:45.091320 9876 consensus_queue.cc:260] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.092518 9876 ts_tablet_manager.cc:1398] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 02:14:45.092859 9876 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:45.092860 9875 tablet_bootstrap.cc:654] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.095373 9876 tablet_bootstrap.cc:654] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.096072 9879 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:45.096274 9879 ts_tablet_manager.cc:1367] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20240503 02:14:45.097126 9879 raft_consensus.cc:348] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.101399 9872 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:45.101892 9872 ts_tablet_manager.cc:1367] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.010s
I20240503 02:14:45.103016 9872 raft_consensus.cc:348] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.103451 9879 raft_consensus.cc:374] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.103660 9879 raft_consensus.cc:729] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.104053 9879 consensus_queue.cc:260] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.104549 9872 raft_consensus.cc:374] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.104808 9872 raft_consensus.cc:729] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.105173 9877 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:45.105387 9877 ts_tablet_manager.cc:1367] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.008s
I20240503 02:14:45.105427 9872 consensus_queue.cc:260] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.106896 9872 ts_tablet_manager.cc:1398] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:14:45.107774 9872 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:45.110226 9872 tablet_bootstrap.cc:654] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.110747 9879 ts_tablet_manager.cc:1398] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:14:45.111111 9879 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:45.106314 9877 raft_consensus.cc:348] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.120038 9877 raft_consensus.cc:374] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.120144 9877 raft_consensus.cc:729] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.120456 9877 consensus_queue.cc:260] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.121264 9877 ts_tablet_manager.cc:1398] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240503 02:14:45.121649 9877 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:45.123641 9877 tablet_bootstrap.cc:654] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.124281 9874 ts_tablet_manager.cc:1398] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.062s user 0.001s sys 0.001s
I20240503 02:14:45.124513 9879 tablet_bootstrap.cc:654] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.128836 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0464bacc4a4bd39dde43ccf4d9a0fc" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:45.129170 9214 raft_consensus.cc:2454] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:45.134128 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0464bacc4a4bd39dde43ccf4d9a0fc" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
W20240503 02:14:45.134968 8492 leader_election.cc:341] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:45.143666 9874 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:45.145372 9874 tablet_bootstrap.cc:654] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.150959 9873 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:45.151230 9873 ts_tablet_manager.cc:1367] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.092s user 0.001s sys 0.016s
I20240503 02:14:45.152256 9873 raft_consensus.cc:348] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.152793 9873 raft_consensus.cc:374] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.152942 9873 raft_consensus.cc:729] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.153178 9875 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:45.153271 9873 consensus_queue.cc:260] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.153796 9875 ts_tablet_manager.cc:1367] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.063s user 0.023s sys 0.009s
I20240503 02:14:45.154726 9876 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:45.154603 9875 raft_consensus.cc:348] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.154929 9876 ts_tablet_manager.cc:1367] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.012s
I20240503 02:14:45.154963 9875 raft_consensus.cc:374] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.155120 9875 raft_consensus.cc:729] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.155431 9875 consensus_queue.cc:260] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.155673 9876 raft_consensus.cc:348] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.155990 9876 raft_consensus.cc:374] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.156131 9876 raft_consensus.cc:729] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.156355 9875 ts_tablet_manager.cc:1398] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:14:45.156395 9876 consensus_queue.cc:260] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.157078 9876 ts_tablet_manager.cc:1398] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:45.157445 9876 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:45.158726 9873 ts_tablet_manager.cc:1398] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240503 02:14:45.160046 9873 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:45.161674 9874 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:45.161857 9874 ts_tablet_manager.cc:1367] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240503 02:14:45.162621 9874 raft_consensus.cc:348] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.162889 9874 raft_consensus.cc:374] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.162961 9874 raft_consensus.cc:729] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.163199 9874 consensus_queue.cc:260] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.163779 9874 ts_tablet_manager.cc:1398] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:14:45.164213 9874 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:45.166150 9874 tablet_bootstrap.cc:654] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.167500 9873 tablet_bootstrap.cc:654] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.168931 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0464bacc4a4bd39dde43ccf4d9a0fc" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:45.169145 8424 raft_consensus.cc:2454] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:45.169255 9879 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:45.169438 9879 ts_tablet_manager.cc:1367] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.058s user 0.000s sys 0.006s
I20240503 02:14:45.169970 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0464bacc4a4bd39dde43ccf4d9a0fc" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:45.170207 8821 raft_consensus.cc:2454] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:45.170223 9879 raft_consensus.cc:348] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.170501 9879 raft_consensus.cc:374] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.170591 9879 raft_consensus.cc:729] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.170850 9879 consensus_queue.cc:260] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.171136 8491 leader_election.cc:302] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, cb90f34dc37b4b16b2a5b5a4ee012865; no voters: 99be4ff1114d4aaabcc537604fce54b3
I20240503 02:14:45.172242 9876 tablet_bootstrap.cc:654] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.172492 9872 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:45.172583 9886 raft_consensus.cc:2781] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:45.172685 9886 raft_consensus.cc:482] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:45.172710 9872 ts_tablet_manager.cc:1367] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.065s user 0.014s sys 0.023s
I20240503 02:14:45.172834 9886 raft_consensus.cc:3037] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.173509 9872 raft_consensus.cc:348] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.173766 9872 raft_consensus.cc:374] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.173844 9872 raft_consensus.cc:729] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.174109 9872 consensus_queue.cc:260] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.174309 9877 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:45.174789 9872 ts_tablet_manager.cc:1398] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:14:45.174985 9877 ts_tablet_manager.cc:1367] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.002s
I20240503 02:14:45.175856 9877 raft_consensus.cc:348] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.176146 9877 raft_consensus.cc:374] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.175904 9886 raft_consensus.cc:504] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.176246 9877 raft_consensus.cc:729] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.177104 9877 consensus_queue.cc:260] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.177356 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0464bacc4a4bd39dde43ccf4d9a0fc" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:45.177577 8950 raft_consensus.cc:3037] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.177634 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0464bacc4a4bd39dde43ccf4d9a0fc" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:45.177708 9877 ts_tablet_manager.cc:1398] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:14:45.177834 9214 raft_consensus.cc:3037] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.178268 9879 ts_tablet_manager.cc:1398] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20240503 02:14:45.178573 9879 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:45.178599 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0464bacc4a4bd39dde43ccf4d9a0fc" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:45.178808 9083 raft_consensus.cc:2454] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:45.180207 9879 tablet_bootstrap.cc:654] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.180414 8950 raft_consensus.cc:2454] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:45.180539 9214 raft_consensus.cc:2454] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:45.181407 9886 leader_election.cc:288] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 election: Requested vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:45.182108 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0464bacc4a4bd39dde43ccf4d9a0fc" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:45.182328 8821 raft_consensus.cc:3037] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.184015 9874 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:45.184159 9874 ts_tablet_manager.cc:1367] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240503 02:14:45.184285 9876 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:45.184417 9876 ts_tablet_manager.cc:1367] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240503 02:14:45.184952 9874 raft_consensus.cc:348] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.185199 9874 raft_consensus.cc:374] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.185288 9874 raft_consensus.cc:729] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.185410 9876 raft_consensus.cc:348] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.185643 9876 raft_consensus.cc:374] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.185534 9874 consensus_queue.cc:260] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.185731 9876 raft_consensus.cc:729] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.186136 9874 ts_tablet_manager.cc:1398] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:14:45.185961 9876 consensus_queue.cc:260] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.186533 9876 ts_tablet_manager.cc:1398] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:14:45.187565 8821 raft_consensus.cc:2454] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:45.189786 9873 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:45.189953 9873 ts_tablet_manager.cc:1367] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.006s
I20240503 02:14:45.190102 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0464bacc4a4bd39dde43ccf4d9a0fc" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:45.190403 8424 raft_consensus.cc:3037] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.190654 9873 raft_consensus.cc:348] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.190860 9873 raft_consensus.cc:374] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.190936 9873 raft_consensus.cc:729] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.191185 9873 consensus_queue.cc:260] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.193343 8491 leader_election.cc:302] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:45.193789 9886 raft_consensus.cc:2781] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:45.193993 9886 raft_consensus.cc:686] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Becoming Leader. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Running, Role: LEADER
I20240503 02:14:45.194101 8424 raft_consensus.cc:2454] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:45.194558 9886 consensus_queue.cc:237] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.194825 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0464bacc4a4bd39dde43ccf4d9a0fc" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:45.195050 9083 raft_consensus.cc:3037] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.197487 9083 raft_consensus.cc:2454] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:45.199995 9879 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:45.200217 9879 ts_tablet_manager.cc:1367] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.006s
I20240503 02:14:45.200237 9873 ts_tablet_manager.cc:1398] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.010s user 0.001s sys 0.002s
I20240503 02:14:45.201313 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0464bacc4a4bd39dde43ccf4d9a0fc" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:45.201546 8689 raft_consensus.cc:2454] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:45.201799 9873 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:45.201877 8688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0464bacc4a4bd39dde43ccf4d9a0fc" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:45.202080 8688 raft_consensus.cc:3037] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.204265 9873 tablet_bootstrap.cc:654] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:45.201066 9879 raft_consensus.cc:348] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.204413 9879 raft_consensus.cc:374] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.204519 9879 raft_consensus.cc:729] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.204787 9879 consensus_queue.cc:260] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.205308 8688 raft_consensus.cc:2454] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:45.206022 9879 ts_tablet_manager.cc:1398] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:14:45.208173 9873 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:45.208374 9873 ts_tablet_manager.cc:1367] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20240503 02:14:45.209481 8280 catalog_manager.cc:5484] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:45.209826 9873 raft_consensus.cc:348] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.210103 9873 raft_consensus.cc:374] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:45.210196 9873 raft_consensus.cc:729] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:45.210415 9873 consensus_queue.cc:260] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.211053 9873 ts_tablet_manager.cc:1398] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:14:45.261191 9886 raft_consensus.cc:482] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:45.261391 9886 raft_consensus.cc:504] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.262547 9886 leader_election.cc:288] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:45.262493 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:45.262751 8821 raft_consensus.cc:2454] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:45.262820 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:45.263075 9214 raft_consensus.cc:2454] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:45.263213 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:45.263324 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:45.263433 8424 raft_consensus.cc:2454] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:45.263427 8688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:45.263573 9083 raft_consensus.cc:2454] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:45.263747 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:45.263850 8491 leader_election.cc:302] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:45.264000 8950 raft_consensus.cc:2454] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:45.264111 9886 raft_consensus.cc:2781] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:45.263659 8688 raft_consensus.cc:2454] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:45.264262 9886 raft_consensus.cc:482] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:45.264357 9886 raft_consensus.cc:3037] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.266263 9886 raft_consensus.cc:504] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.267246 9886 leader_election.cc:288] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 election: Requested vote from peers 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:45.267397 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:45.267396 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:45.267519 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:45.267576 9083 raft_consensus.cc:3037] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.267616 8821 raft_consensus.cc:3037] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.267694 9214 raft_consensus.cc:3037] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.269637 9214 raft_consensus.cc:2454] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:45.270371 8821 raft_consensus.cc:2454] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:45.270388 9083 raft_consensus.cc:2454] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:45.270397 8688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:45.270656 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:45.270673 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e42dca2db54de5a9417df20242e2c2" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:45.270699 8688 raft_consensus.cc:3037] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.270851 8950 raft_consensus.cc:3037] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.270843 8424 raft_consensus.cc:3037] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.272161 8491 leader_election.cc:302] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 79e41f25066e4e5abbd71e48d545c6bf, c399efcbba924863ac89d156dc07ac7d, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:45.272503 9886 raft_consensus.cc:2781] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:45.272673 9886 raft_consensus.cc:686] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Becoming Leader. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Running, Role: LEADER
I20240503 02:14:45.273160 8424 raft_consensus.cc:2454] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:45.273051 9886 consensus_queue.cc:237] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.273528 8950 raft_consensus.cc:2454] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:45.274080 8688 raft_consensus.cc:2454] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:45.276528 8279 catalog_manager.cc:5484] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:45.288110 9792 raft_consensus.cc:482] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:45.288259 9792 raft_consensus.cc:504] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.289400 9792 leader_election.cc:288] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:45.289381 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71357f9f8ef449189f485b3246869f1" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:45.289616 8950 raft_consensus.cc:2454] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:45.289881 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71357f9f8ef449189f485b3246869f1" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:45.289942 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71357f9f8ef449189f485b3246869f1" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:45.289948 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71357f9f8ef449189f485b3246869f1" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:45.290107 9214 raft_consensus.cc:2454] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:45.290155 8424 raft_consensus.cc:2454] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:45.290165 8821 raft_consensus.cc:2454] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:45.290658 8557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71357f9f8ef449189f485b3246869f1" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:45.290663 9019 leader_election.cc:302] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 5 yes votes; 0 no votes. yes voters: 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:45.290724 8688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71357f9f8ef449189f485b3246869f1" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:45.290889 8557 raft_consensus.cc:2454] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:45.290940 8688 raft_consensus.cc:2454] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:45.290989 9792 raft_consensus.cc:2781] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:45.291095 9792 raft_consensus.cc:482] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:45.291186 9792 raft_consensus.cc:3037] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.293016 9792 raft_consensus.cc:504] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.294154 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71357f9f8ef449189f485b3246869f1" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:45.294169 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71357f9f8ef449189f485b3246869f1" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:45.294214 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71357f9f8ef449189f485b3246869f1" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:45.294274 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71357f9f8ef449189f485b3246869f1" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:45.294325 8821 raft_consensus.cc:3037] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.294353 8950 raft_consensus.cc:3037] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.294399 8424 raft_consensus.cc:3037] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.294435 9214 raft_consensus.cc:3037] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.297144 8950 raft_consensus.cc:2454] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:45.297158 8821 raft_consensus.cc:2454] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:45.297181 8424 raft_consensus.cc:2454] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:45.297181 9214 raft_consensus.cc:2454] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:45.297492 8557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71357f9f8ef449189f485b3246869f1" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:45.297717 8557 raft_consensus.cc:3037] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.297839 9017 leader_election.cc:302] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d; no voters:
I20240503 02:14:45.298653 9792 leader_election.cc:288] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 election: Requested vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:45.298882 9792 raft_consensus.cc:2781] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:45.299023 9792 raft_consensus.cc:686] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Becoming Leader. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Running, Role: LEADER
I20240503 02:14:45.300216 8557 raft_consensus.cc:2454] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:45.300024 9792 consensus_queue.cc:237] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:45.301470 8688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71357f9f8ef449189f485b3246869f1" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:45.301715 8688 raft_consensus.cc:3037] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:45.304791 8688 raft_consensus.cc:2454] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:45.304649 8279 catalog_manager.cc:5484] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c399efcbba924863ac89d156dc07ac7d (127.23.104.198). New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:45.583943 9886 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:45.636021 9886 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:45.686818 9886 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:45.692840 9886 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:45.696619 9920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:45.696969 9920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:14:45.703097 9792 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:45.709321 9920 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:14:45.711199 9886 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:45.717270 9882 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:45.733928 9792 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:45.743049 9882 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240503 02:14:45.788000 9792 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:45.793840 9963 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:45.796386 9917 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:45.808185 9886 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:45.810246 9932 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:45.818223 9963 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:45.826257 9932 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:45.856670 9932 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:45.866303 9932 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 536.308 ms
time per row: 0.268154 ms
I20240503 02:14:46.407886 8278 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42220:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:14:46.408514 8278 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 02:14:46.416603 9061 tablet_service.cc:1467] Processing CreateTablet for tablet df68cc6032b14cae9e1cac074976baf8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.416787 8930 tablet_service.cc:1467] Processing CreateTablet for tablet df68cc6032b14cae9e1cac074976baf8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.417062 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df68cc6032b14cae9e1cac074976baf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.417189 8930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df68cc6032b14cae9e1cac074976baf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.417250 9194 tablet_service.cc:1467] Processing CreateTablet for tablet df68cc6032b14cae9e1cac074976baf8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.417627 9194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df68cc6032b14cae9e1cac074976baf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.418138 8800 tablet_service.cc:1467] Processing CreateTablet for tablet df68cc6032b14cae9e1cac074976baf8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.418529 8800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df68cc6032b14cae9e1cac074976baf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.423463 8932 tablet_service.cc:1467] Processing CreateTablet for tablet 493e2b13e19241fa8fa48c367986caaf (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.423981 8932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493e2b13e19241fa8fa48c367986caaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.429617 8669 tablet_service.cc:1467] Processing CreateTablet for tablet df68cc6032b14cae9e1cac074976baf8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.429667 8405 tablet_service.cc:1467] Processing CreateTablet for tablet df68cc6032b14cae9e1cac074976baf8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.430078 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df68cc6032b14cae9e1cac074976baf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.430181 8669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df68cc6032b14cae9e1cac074976baf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.432396 10016 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:46.433763 9062 tablet_service.cc:1467] Processing CreateTablet for tablet 493e2b13e19241fa8fa48c367986caaf (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.434165 8801 tablet_service.cc:1467] Processing CreateTablet for tablet 493e2b13e19241fa8fa48c367986caaf (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.434276 9062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493e2b13e19241fa8fa48c367986caaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.434542 8801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493e2b13e19241fa8fa48c367986caaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.436691 10016 tablet_bootstrap.cc:654] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.440677 8537 tablet_service.cc:1467] Processing CreateTablet for tablet df68cc6032b14cae9e1cac074976baf8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.441200 8537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df68cc6032b14cae9e1cac074976baf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.442461 9193 tablet_service.cc:1467] Processing CreateTablet for tablet 493e2b13e19241fa8fa48c367986caaf (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.442857 9193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493e2b13e19241fa8fa48c367986caaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.446931 8670 tablet_service.cc:1467] Processing CreateTablet for tablet 493e2b13e19241fa8fa48c367986caaf (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.447202 8403 tablet_service.cc:1467] Processing CreateTablet for tablet fe35d536ab6348c58e90d680383bb763 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.447312 8670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493e2b13e19241fa8fa48c367986caaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.447572 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe35d536ab6348c58e90d680383bb763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.448081 8404 tablet_service.cc:1467] Processing CreateTablet for tablet 493e2b13e19241fa8fa48c367986caaf (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.448475 8404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493e2b13e19241fa8fa48c367986caaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.451550 8800 tablet_service.cc:1467] Processing CreateTablet for tablet fe35d536ab6348c58e90d680383bb763 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.452389 9063 tablet_service.cc:1467] Processing CreateTablet for tablet fe35d536ab6348c58e90d680383bb763 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.452755 9063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe35d536ab6348c58e90d680383bb763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.452872 8931 tablet_service.cc:1467] Processing CreateTablet for tablet fe35d536ab6348c58e90d680383bb763 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.453248 8931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe35d536ab6348c58e90d680383bb763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.453290 8800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe35d536ab6348c58e90d680383bb763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.453696 9192 tablet_service.cc:1467] Processing CreateTablet for tablet fe35d536ab6348c58e90d680383bb763 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.454046 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe35d536ab6348c58e90d680383bb763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.456158 8668 tablet_service.cc:1467] Processing CreateTablet for tablet fe35d536ab6348c58e90d680383bb763 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.456527 8668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe35d536ab6348c58e90d680383bb763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.460608 8536 tablet_service.cc:1467] Processing CreateTablet for tablet 493e2b13e19241fa8fa48c367986caaf (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.461109 8536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493e2b13e19241fa8fa48c367986caaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.464255 8538 tablet_service.cc:1467] Processing CreateTablet for tablet fe35d536ab6348c58e90d680383bb763 (DEFAULT_TABLE table=rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:46.464699 8538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe35d536ab6348c58e90d680383bb763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:46.466986 10017 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:46.468009 10019 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:46.469601 10019 tablet_bootstrap.cc:654] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.470625 10017 tablet_bootstrap.cc:654] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.472040 10022 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:46.472040 10014 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:46.473758 10014 tablet_bootstrap.cc:654] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.476415 10022 tablet_bootstrap.cc:654] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.478390 10014 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:46.478549 10014 ts_tablet_manager.cc:1367] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:14:46.479452 10014 raft_consensus.cc:348] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.479822 10014 raft_consensus.cc:374] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.479931 10016 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:46.479928 10014 raft_consensus.cc:729] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.480095 10016 ts_tablet_manager.cc:1367] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.048s user 0.020s sys 0.000s
I20240503 02:14:46.480506 10019 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:46.480669 10019 ts_tablet_manager.cc:1367] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20240503 02:14:46.480896 10016 raft_consensus.cc:348] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.481194 10016 raft_consensus.cc:374] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.481307 10016 raft_consensus.cc:729] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.481565 10016 consensus_queue.cc:260] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.482060 10015 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:46.482237 10017 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:46.482455 10017 ts_tablet_manager.cc:1367] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20240503 02:14:46.483282 10017 raft_consensus.cc:348] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.483554 10017 raft_consensus.cc:374] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.483661 10017 raft_consensus.cc:729] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.483875 10015 tablet_bootstrap.cc:654] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.483980 10017 consensus_queue.cc:260] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.484854 10016 ts_tablet_manager.cc:1398] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:14:46.484772 10019 raft_consensus.cc:348] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.485289 10019 raft_consensus.cc:374] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.485407 10019 raft_consensus.cc:729] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.485785 10019 consensus_queue.cc:260] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.486258 10014 consensus_queue.cc:260] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.486949 10016 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:46.487143 10019 ts_tablet_manager.cc:1398] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:14:46.488483 10016 tablet_bootstrap.cc:654] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.488473 10015 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:46.488643 10015 ts_tablet_manager.cc:1367] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240503 02:14:46.491075 10015 raft_consensus.cc:348] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.491469 10019 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:46.493106 10019 tablet_bootstrap.cc:654] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.496176 10014 ts_tablet_manager.cc:1398] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20240503 02:14:46.496667 10017 ts_tablet_manager.cc:1398] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20240503 02:14:46.497027 10017 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:46.498224 10030 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:46.499630 10017 tablet_bootstrap.cc:654] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.501581 10030 tablet_bootstrap.cc:654] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.502421 10014 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:46.505457 10016 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:46.505650 10016 ts_tablet_manager.cc:1367] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.000s
I20240503 02:14:46.506290 10014 tablet_bootstrap.cc:654] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.506356 10016 raft_consensus.cc:348] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.506652 10016 raft_consensus.cc:374] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.506737 10016 raft_consensus.cc:729] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.506800 10019 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:46.506985 10019 ts_tablet_manager.cc:1367] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240503 02:14:46.507042 10016 consensus_queue.cc:260] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.507717 10019 raft_consensus.cc:348] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.508036 10019 raft_consensus.cc:374] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.508143 10019 raft_consensus.cc:729] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.508404 10015 raft_consensus.cc:374] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.508399 10019 consensus_queue.cc:260] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.509084 10019 ts_tablet_manager.cc:1398] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:46.509416 10019 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:46.508726 10015 raft_consensus.cc:729] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.510447 10016 ts_tablet_manager.cc:1398] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:14:46.509255 10017 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:46.510891 10019 tablet_bootstrap.cc:654] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.510892 10017 ts_tablet_manager.cc:1367] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 02:14:46.511416 10016 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:46.512959 10022 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:46.513309 10022 ts_tablet_manager.cc:1367] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.003s
I20240503 02:14:46.513367 10016 tablet_bootstrap.cc:654] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.513936 10017 raft_consensus.cc:348] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.514250 10017 raft_consensus.cc:374] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.514341 10017 raft_consensus.cc:729] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.514289 10022 raft_consensus.cc:348] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.514695 10022 raft_consensus.cc:374] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.514585 10017 consensus_queue.cc:260] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.514873 10022 raft_consensus.cc:729] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.515167 10017 ts_tablet_manager.cc:1398] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:14:46.515167 10022 consensus_queue.cc:260] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.515420 10030 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:46.515463 10017 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:46.515615 10030 ts_tablet_manager.cc:1367] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.004s
I20240503 02:14:46.515923 10015 consensus_queue.cc:260] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.516906 10017 tablet_bootstrap.cc:654] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.516610 10030 raft_consensus.cc:348] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.517010 10030 raft_consensus.cc:374] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.517124 10030 raft_consensus.cc:729] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.517426 10030 consensus_queue.cc:260] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.519071 10015 ts_tablet_manager.cc:1398] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240503 02:14:46.520151 10015 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:46.523275 10030 ts_tablet_manager.cc:1398] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240503 02:14:46.523617 10030 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:46.524343 10014 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:46.524624 10014 ts_tablet_manager.cc:1367] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240503 02:14:46.525172 10030 tablet_bootstrap.cc:654] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.525658 10014 raft_consensus.cc:348] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.526342 10014 raft_consensus.cc:374] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.526492 10014 raft_consensus.cc:729] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.528465 10019 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:46.528642 10019 ts_tablet_manager.cc:1367] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.010s
I20240503 02:14:46.528369 10014 consensus_queue.cc:260] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.529253 10014 ts_tablet_manager.cc:1398] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:14:46.529599 10014 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:46.529528 10019 raft_consensus.cc:348] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.529865 10019 raft_consensus.cc:374] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.530016 10019 raft_consensus.cc:729] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.530350 10019 consensus_queue.cc:260] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.531289 10019 ts_tablet_manager.cc:1398] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 02:14:46.532002 10014 tablet_bootstrap.cc:654] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.533668 10030 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:46.533823 10030 ts_tablet_manager.cc:1367] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20240503 02:14:46.534719 10014 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:46.534533 10030 raft_consensus.cc:348] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.534729 10015 tablet_bootstrap.cc:654] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.534811 10030 raft_consensus.cc:374] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.534888 10030 raft_consensus.cc:729] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.535018 10014 ts_tablet_manager.cc:1367] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240503 02:14:46.535171 10030 consensus_queue.cc:260] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.535851 10030 ts_tablet_manager.cc:1398] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:14:46.536217 10030 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:46.536008 10014 raft_consensus.cc:348] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.536695 10014 raft_consensus.cc:374] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.536868 10014 raft_consensus.cc:729] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.537168 10014 consensus_queue.cc:260] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.537809 10030 tablet_bootstrap.cc:654] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.537921 10014 ts_tablet_manager.cc:1398] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:14:46.538390 10022 ts_tablet_manager.cc:1398] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.025s user 0.001s sys 0.002s
I20240503 02:14:46.540472 10016 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:46.540620 10016 ts_tablet_manager.cc:1367] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.000s
I20240503 02:14:46.541216 10016 raft_consensus.cc:348] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.541447 10016 raft_consensus.cc:374] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.541553 10016 raft_consensus.cc:729] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.541786 10016 consensus_queue.cc:260] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.541978 10022 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:46.542333 10016 ts_tablet_manager.cc:1398] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:46.543159 10030 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:46.543288 10030 ts_tablet_manager.cc:1367] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:14:46.543489 10022 tablet_bootstrap.cc:654] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.543958 10030 raft_consensus.cc:348] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.544246 10030 raft_consensus.cc:374] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.544363 10030 raft_consensus.cc:729] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.544951 10024 raft_consensus.cc:482] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:46.544677 10030 consensus_queue.cc:260] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.545342 10017 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:46.545140 10024 raft_consensus.cc:504] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.545509 10017 ts_tablet_manager.cc:1367] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.005s
I20240503 02:14:46.545567 10030 ts_tablet_manager.cc:1398] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:46.548971 10017 raft_consensus.cc:348] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.549268 10017 raft_consensus.cc:374] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.549393 10017 raft_consensus.cc:729] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.549669 10017 consensus_queue.cc:260] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.554312 9996 raft_consensus.cc:482] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:46.554513 9996 raft_consensus.cc:504] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.556286 9996 leader_election.cc:288] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:46.557232 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493e2b13e19241fa8fa48c367986caaf" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:46.557500 8950 raft_consensus.cc:2454] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:46.558568 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493e2b13e19241fa8fa48c367986caaf" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:46.558848 9213 raft_consensus.cc:2454] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:46.560124 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493e2b13e19241fa8fa48c367986caaf" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:46.560274 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493e2b13e19241fa8fa48c367986caaf" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:46.560397 8821 raft_consensus.cc:2454] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
W20240503 02:14:46.561573 8491 leader_election.cc:341] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:46.561822 8491 leader_election.cc:302] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865; no voters: c399efcbba924863ac89d156dc07ac7d
I20240503 02:14:46.562263 10024 leader_election.cc:288] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:46.562510 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493e2b13e19241fa8fa48c367986caaf" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
W20240503 02:14:46.564102 8491 leader_election.cc:341] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:46.564394 9996 raft_consensus.cc:2781] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:46.564522 9996 raft_consensus.cc:482] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:46.564633 9996 raft_consensus.cc:3037] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.568714 9996 raft_consensus.cc:504] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.569541 10017 ts_tablet_manager.cc:1398] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.024s user 0.002s sys 0.001s
I20240503 02:14:46.570441 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493e2b13e19241fa8fa48c367986caaf" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:46.570715 8950 raft_consensus.cc:3037] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.572274 10022 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:46.572494 10022 ts_tablet_manager.cc:1367] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.003s
I20240503 02:14:46.574293 10022 raft_consensus.cc:348] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.574640 10022 raft_consensus.cc:374] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.574627 8950 raft_consensus.cc:2454] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:46.574713 10022 raft_consensus.cc:729] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.574972 10022 consensus_queue.cc:260] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.576135 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493e2b13e19241fa8fa48c367986caaf" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:46.576520 10022 ts_tablet_manager.cc:1398] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 02:14:46.576853 10022 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
W20240503 02:14:46.576942 8491 leader_election.cc:341] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:46.578056 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493e2b13e19241fa8fa48c367986caaf" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:46.578272 8821 raft_consensus.cc:3037] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.578174 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493e2b13e19241fa8fa48c367986caaf" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:46.578433 9213 raft_consensus.cc:3037] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.581238 8821 raft_consensus.cc:2454] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:46.582855 9213 raft_consensus.cc:2454] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:46.583554 8493 leader_election.cc:302] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865; no voters: c399efcbba924863ac89d156dc07ac7d
I20240503 02:14:46.584146 9932 raft_consensus.cc:2781] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:46.584306 9932 raft_consensus.cc:686] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Becoming Leader. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Running, Role: LEADER
I20240503 02:14:46.584805 9932 consensus_queue.cc:237] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.589432 10022 tablet_bootstrap.cc:654] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.591895 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493e2b13e19241fa8fa48c367986caaf" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:46.592154 8689 raft_consensus.cc:3037] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.596143 8689 raft_consensus.cc:2454] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:46.596935 9996 leader_election.cc:288] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 election: Requested vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:46.601799 10022 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:46.601954 10022 ts_tablet_manager.cc:1367] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20240503 02:14:46.602800 10022 raft_consensus.cc:348] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.603114 10022 raft_consensus.cc:374] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.603209 10022 raft_consensus.cc:729] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.603479 10022 consensus_queue.cc:260] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.604202 10022 ts_tablet_manager.cc:1398] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:46.608721 10036 raft_consensus.cc:482] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:46.608887 10036 raft_consensus.cc:504] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.610074 10036 leader_election.cc:288] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:46.610774 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:46.610992 8558 raft_consensus.cc:2454] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 0.
I20240503 02:14:46.610850 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:46.611302 9213 raft_consensus.cc:2454] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 0.
I20240503 02:14:46.612200 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:46.612485 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:46.612692 8821 raft_consensus.cc:2454] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 0.
I20240503 02:14:46.613049 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:46.613421 8950 raft_consensus.cc:2454] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 0.
I20240503 02:14:46.613937 8624 leader_election.cc:302] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 846e89dd0b154d2b9bb7d4f315c9fc12, 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:46.614354 10036 raft_consensus.cc:2781] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:46.614490 10036 raft_consensus.cc:482] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:46.614616 10036 raft_consensus.cc:3037] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.615778 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:46.616020 8425 raft_consensus.cc:2454] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 0.
W20240503 02:14:46.616539 8623 leader_election.cc:341] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:46.620965 10036 raft_consensus.cc:504] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.623529 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:46.623775 8558 raft_consensus.cc:3037] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.624226 10053 raft_consensus.cc:482] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:46.624392 10053 raft_consensus.cc:504] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.625792 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:46.626011 9213 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 0.
I20240503 02:14:46.627532 8558 raft_consensus.cc:2454] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 1.
I20240503 02:14:46.631968 10036 leader_election.cc:288] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 election: Requested vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:46.632405 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:46.632737 9213 raft_consensus.cc:3037] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.633141 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493e2b13e19241fa8fa48c367986caaf" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:46.633366 8425 raft_consensus.cc:2454] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:46.633744 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493e2b13e19241fa8fa48c367986caaf" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:46.633929 8425 raft_consensus.cc:3037] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.635084 9995 raft_consensus.cc:482] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:46.635300 9995 raft_consensus.cc:504] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.637238 8425 raft_consensus.cc:2454] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:46.637529 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:46.637723 8821 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:46.637986 10027 raft_consensus.cc:482] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:46.638155 9995 leader_election.cc:288] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:46.638155 10027 raft_consensus.cc:504] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.638849 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:46.639104 8689 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:46.639406 10027 leader_election.cc:288] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:46.639919 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:46.640180 8950 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:46.640620 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:46.640827 8821 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:46.641120 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:46.641342 9214 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:46.641470 9018 leader_election.cc:302] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 79e41f25066e4e5abbd71e48d545c6bf, 846e89dd0b154d2b9bb7d4f315c9fc12, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d; no voters:
I20240503 02:14:46.642002 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:46.642123 9995 raft_consensus.cc:2781] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:46.642283 9082 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:46.642412 9995 raft_consensus.cc:482] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:46.642539 9995 raft_consensus.cc:3037] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.643064 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:46.643404 8558 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:46.644742 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:46.644908 8425 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:46.648499 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:46.648697 8558 raft_consensus.cc:2379] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c668b2e2d5c4edba46cbc5c36065c0c in current term 1: Already voted for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in this term.
I20240503 02:14:46.649220 9995 raft_consensus.cc:504] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.651296 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:46.651468 8821 raft_consensus.cc:3037] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.652457 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:46.652706 8558 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 0.
I20240503 02:14:46.653225 10053 leader_election.cc:288] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923)
I20240503 02:14:46.654992 8821 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:46.655958 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:46.656126 9214 raft_consensus.cc:3037] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.657444 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:46.657487 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:46.657621 8425 raft_consensus.cc:3037] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.657685 8558 raft_consensus.cc:3037] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.658169 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:46.660641 8425 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:46.662050 8558 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:46.662982 9019 leader_election.cc:302] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, c399efcbba924863ac89d156dc07ac7d; no voters:
I20240503 02:14:46.663285 9994 raft_consensus.cc:2781] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:46.663437 9994 raft_consensus.cc:686] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Becoming Leader. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Running, Role: LEADER
I20240503 02:14:46.663836 9994 consensus_queue.cc:237] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.666085 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:46.666236 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:46.666441 8689 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
W20240503 02:14:46.666760 8358 leader_election.cc:341] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:46.667021 8886 leader_election.cc:302] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 79e41f25066e4e5abbd71e48d545c6bf, 846e89dd0b154d2b9bb7d4f315c9fc12, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d; no voters: 4c668b2e2d5c4edba46cbc5c36065c0c
I20240503 02:14:46.667443 10027 raft_consensus.cc:2781] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:46.667585 10027 raft_consensus.cc:482] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:46.667711 10027 raft_consensus.cc:3037] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.669262 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:46.671255 10027 raft_consensus.cc:504] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.671577 9995 leader_election.cc:288] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 election: Requested vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:46.672210 8950 raft_consensus.cc:2379] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c399efcbba924863ac89d156dc07ac7d in current term 1: Already voted for candidate 99be4ff1114d4aaabcc537604fce54b3 in this term.
I20240503 02:14:46.672778 10027 leader_election.cc:288] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 election: Requested vote from peers c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:46.675712 9213 raft_consensus.cc:2454] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 1.
I20240503 02:14:46.676430 9214 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:46.676651 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:46.676908 8558 raft_consensus.cc:2379] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99be4ff1114d4aaabcc537604fce54b3 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
I20240503 02:14:46.677804 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:46.678118 8821 raft_consensus.cc:2379] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99be4ff1114d4aaabcc537604fce54b3 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
I20240503 02:14:46.678020 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:46.678344 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:46.678547 8689 raft_consensus.cc:3037] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.679461 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:46.679553 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:46.680083 8425 raft_consensus.cc:2379] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99be4ff1114d4aaabcc537604fce54b3 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
I20240503 02:14:46.682016 8689 raft_consensus.cc:2454] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:46.682312 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:46.682555 8558 raft_consensus.cc:2379] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99be4ff1114d4aaabcc537604fce54b3 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
I20240503 02:14:46.683575 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:46.683868 8950 raft_consensus.cc:2379] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in current term 1: Already voted for candidate 99be4ff1114d4aaabcc537604fce54b3 in this term.
I20240503 02:14:46.684762 8952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:46.684914 8952 raft_consensus.cc:3037] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.687487 8952 raft_consensus.cc:2454] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 1.
I20240503 02:14:46.693192 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:46.693502 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:46.693599 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:46.693802 8425 raft_consensus.cc:3037] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.694257 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:46.694547 8690 raft_consensus.cc:2379] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c668b2e2d5c4edba46cbc5c36065c0c in current term 1: Already voted for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in this term.
I20240503 02:14:46.694944 8952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:46.695173 8952 raft_consensus.cc:2379] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c668b2e2d5c4edba46cbc5c36065c0c in current term 1: Already voted for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in this term.
I20240503 02:14:46.695533 8888 leader_election.cc:302] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 846e89dd0b154d2b9bb7d4f315c9fc12, 99be4ff1114d4aaabcc537604fce54b3; no voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, c399efcbba924863ac89d156dc07ac7d
I20240503 02:14:46.695847 10027 raft_consensus.cc:2726] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:14:46.696871 10015 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:46.697132 10015 ts_tablet_manager.cc:1367] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.177s user 0.007s sys 0.006s
I20240503 02:14:46.697104 8820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:46.697388 8820 raft_consensus.cc:3037] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.698314 10015 raft_consensus.cc:348] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
W20240503 02:14:46.699743 8623 leader_election.cc:341] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:46.699925 8625 leader_election.cc:302] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 846e89dd0b154d2b9bb7d4f315c9fc12, 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865; no voters: c399efcbba924863ac89d156dc07ac7d
I20240503 02:14:46.701843 8359 leader_election.cc:302] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 4c668b2e2d5c4edba46cbc5c36065c0c; no voters: 2a1893ab239f4163a06bdbf8ce579431, 846e89dd0b154d2b9bb7d4f315c9fc12, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d
I20240503 02:14:46.702886 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:46.703224 8424 raft_consensus.cc:2379] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
I20240503 02:14:46.704507 8425 raft_consensus.cc:2454] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 1.
I20240503 02:14:46.704699 10048 raft_consensus.cc:2726] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:46.706264 10015 raft_consensus.cc:374] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.706640 10015 raft_consensus.cc:729] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.706995 10015 consensus_queue.cc:260] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.707829 10053 raft_consensus.cc:2781] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:46.708063 10053 raft_consensus.cc:686] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 LEADER]: Becoming Leader. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Running, Role: LEADER
I20240503 02:14:46.707950 8280 catalog_manager.cc:5484] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:46.707770 10015 ts_tablet_manager.cc:1398] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:14:46.708951 10015 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:46.708772 10053 consensus_queue.cc:237] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.710580 8282 catalog_manager.cc:5484] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c399efcbba924863ac89d156dc07ac7d (127.23.104.198). New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:46.713223 8820 raft_consensus.cc:2454] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 1.
I20240503 02:14:46.713779 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:46.714035 8821 raft_consensus.cc:2379] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
I20240503 02:14:46.714468 8623 leader_election.cc:302] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 846e89dd0b154d2b9bb7d4f315c9fc12, cb90f34dc37b4b16b2a5b5a4ee012865; no voters: 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d
I20240503 02:14:46.714957 10036 raft_consensus.cc:2726] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:46.717854 8280 catalog_manager.cc:5484] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195). New cstate: current_term: 1 leader_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:46.720144 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:46.722157 9213 raft_consensus.cc:2379] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99be4ff1114d4aaabcc537604fce54b3 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
I20240503 02:14:46.722522 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df68cc6032b14cae9e1cac074976baf8" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:46.722769 9214 raft_consensus.cc:2379] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99be4ff1114d4aaabcc537604fce54b3 in current term 1: Already voted for candidate c399efcbba924863ac89d156dc07ac7d in this term.
I20240503 02:14:46.724525 10015 tablet_bootstrap.cc:654] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:46.728567 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:46.728893 9214 raft_consensus.cc:2379] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c668b2e2d5c4edba46cbc5c36065c0c in current term 1: Already voted for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in this term.
I20240503 02:14:46.729355 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:46.729581 8821 raft_consensus.cc:2379] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c668b2e2d5c4edba46cbc5c36065c0c in current term 1: Already voted for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in this term.
I20240503 02:14:46.730772 10015 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:46.730901 10015 ts_tablet_manager.cc:1367] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.004s
I20240503 02:14:46.731457 10015 raft_consensus.cc:348] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.731716 10015 raft_consensus.cc:374] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:46.731848 10015 raft_consensus.cc:729] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:46.732180 10015 consensus_queue.cc:260] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:46.732726 10015 ts_tablet_manager.cc:1398] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:14:46.742342 9994 raft_consensus.cc:482] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:46.742547 9994 raft_consensus.cc:504] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:46.743746 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:46.743932 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:46.744027 8425 raft_consensus.cc:2379] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c399efcbba924863ac89d156dc07ac7d in current term 1: Already voted for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in this term.
I20240503 02:14:46.744176 8558 raft_consensus.cc:2379] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c399efcbba924863ac89d156dc07ac7d in current term 1: Already voted for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in this term.
I20240503 02:14:46.744155 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:46.744550 9994 leader_election.cc:288] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:46.744699 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:46.744768 8952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:46.744957 9214 raft_consensus.cc:2379] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c399efcbba924863ac89d156dc07ac7d in current term 1: Already voted for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in this term.
I20240503 02:14:46.745030 8952 raft_consensus.cc:2379] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c399efcbba924863ac89d156dc07ac7d in current term 1: Already voted for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in this term.
I20240503 02:14:46.745333 9019 leader_election.cc:302] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: c399efcbba924863ac89d156dc07ac7d; no voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 846e89dd0b154d2b9bb7d4f315c9fc12, cb90f34dc37b4b16b2a5b5a4ee012865
I20240503 02:14:46.745616 9994 raft_consensus.cc:3037] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:46.746107 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe35d536ab6348c58e90d680383bb763" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:46.746377 8821 raft_consensus.cc:2379] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c399efcbba924863ac89d156dc07ac7d in current term 1: Already voted for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in this term.
I20240503 02:14:46.748137 9994 raft_consensus.cc:2726] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:46.971319 9932 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:46.981966 9932 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:47.052196 9083 raft_consensus.cc:3037] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:47.055868 9932 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:47.057965 9932 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:47.061479 9994 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:47.070675 9996 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:47.089938 9994 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:47.094161 9972 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:47.097493 10053 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:47.103154 10053 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:47.126319 9994 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:47.130044 9994 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:47.135879 10053 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:47.184679 9995 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:47.203373 9994 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:47.212159 10036 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:47.233558 10059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:47.234016 10059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:14:47.244007 10036 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:47.245787 10059 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:14:47.274551 10036 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
W20240503 02:14:47.517264 9019 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:45923, user_credentials={real_user=slave}} blocked reactor thread for 74230.6us
INSERT report
rows total: 2000
time total: 438.871 ms
time per row: 0.219435 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:14:48.802134 8282 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42220:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:14:48.802831 8282 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_3 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 02:14:48.811501 8405 tablet_service.cc:1467] Processing CreateTablet for tablet 64c52f9977ae4eb4ac803be99b6426c2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.811973 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c52f9977ae4eb4ac803be99b6426c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.812402 8668 tablet_service.cc:1467] Processing CreateTablet for tablet 64c52f9977ae4eb4ac803be99b6426c2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.812507 8800 tablet_service.cc:1467] Processing CreateTablet for tablet 64c52f9977ae4eb4ac803be99b6426c2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.812858 8668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c52f9977ae4eb4ac803be99b6426c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.812950 8800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c52f9977ae4eb4ac803be99b6426c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.815868 8536 tablet_service.cc:1467] Processing CreateTablet for tablet 64c52f9977ae4eb4ac803be99b6426c2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.816211 8536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c52f9977ae4eb4ac803be99b6426c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.819069 8930 tablet_service.cc:1467] Processing CreateTablet for tablet 64c52f9977ae4eb4ac803be99b6426c2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.819480 8930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c52f9977ae4eb4ac803be99b6426c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.820628 8404 tablet_service.cc:1467] Processing CreateTablet for tablet 95a8f2627c5c49a39291c96eb00fec18 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.820658 9194 tablet_service.cc:1467] Processing CreateTablet for tablet 64c52f9977ae4eb4ac803be99b6426c2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.821002 8404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a8f2627c5c49a39291c96eb00fec18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.821062 9194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c52f9977ae4eb4ac803be99b6426c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.824124 9193 tablet_service.cc:1467] Processing CreateTablet for tablet 95a8f2627c5c49a39291c96eb00fec18 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.824477 9193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a8f2627c5c49a39291c96eb00fec18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.824481 8801 tablet_service.cc:1467] Processing CreateTablet for tablet 95a8f2627c5c49a39291c96eb00fec18 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.824525 8403 tablet_service.cc:1467] Processing CreateTablet for tablet 3bca5fd721f84f38bff08308965eb33f (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.824841 8801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a8f2627c5c49a39291c96eb00fec18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.824885 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bca5fd721f84f38bff08308965eb33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.828096 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 3bca5fd721f84f38bff08308965eb33f (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.828130 8799 tablet_service.cc:1467] Processing CreateTablet for tablet 3bca5fd721f84f38bff08308965eb33f (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.828410 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bca5fd721f84f38bff08308965eb33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.828466 8799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bca5fd721f84f38bff08308965eb33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.829087 9063 tablet_service.cc:1467] Processing CreateTablet for tablet 64c52f9977ae4eb4ac803be99b6426c2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.829469 9063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c52f9977ae4eb4ac803be99b6426c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.831682 8932 tablet_service.cc:1467] Processing CreateTablet for tablet 95a8f2627c5c49a39291c96eb00fec18 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.832114 8932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a8f2627c5c49a39291c96eb00fec18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.832170 9062 tablet_service.cc:1467] Processing CreateTablet for tablet 95a8f2627c5c49a39291c96eb00fec18 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.832520 9062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a8f2627c5c49a39291c96eb00fec18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.835520 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 3bca5fd721f84f38bff08308965eb33f (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.835594 8931 tablet_service.cc:1467] Processing CreateTablet for tablet 3bca5fd721f84f38bff08308965eb33f (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.835918 8931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bca5fd721f84f38bff08308965eb33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.836688 8538 tablet_service.cc:1467] Processing CreateTablet for tablet 3bca5fd721f84f38bff08308965eb33f (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.837169 8538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bca5fd721f84f38bff08308965eb33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.838830 10166 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:48.841614 8537 tablet_service.cc:1467] Processing CreateTablet for tablet 95a8f2627c5c49a39291c96eb00fec18 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.842039 8537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a8f2627c5c49a39291c96eb00fec18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.842500 10166 tablet_bootstrap.cc:654] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.844949 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bca5fd721f84f38bff08308965eb33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.845232 8669 tablet_service.cc:1467] Processing CreateTablet for tablet 95a8f2627c5c49a39291c96eb00fec18 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.845723 8669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a8f2627c5c49a39291c96eb00fec18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.848125 8670 tablet_service.cc:1467] Processing CreateTablet for tablet 3bca5fd721f84f38bff08308965eb33f (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:48.848460 8670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bca5fd721f84f38bff08308965eb33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:48.859205 10172 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:48.863473 10172 tablet_bootstrap.cc:654] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.869028 10173 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:48.870416 10169 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:48.870880 10173 tablet_bootstrap.cc:654] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.872473 10169 tablet_bootstrap.cc:654] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.873440 10168 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:48.875125 10168 tablet_bootstrap.cc:654] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.876037 10169 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:48.876384 10169 ts_tablet_manager.cc:1367] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:14:48.876706 10172 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:48.876953 10172 ts_tablet_manager.cc:1367] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.005s
I20240503 02:14:48.877099 10171 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:48.877306 10169 raft_consensus.cc:348] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.877703 10169 raft_consensus.cc:374] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.877890 10169 raft_consensus.cc:729] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.877876 10172 raft_consensus.cc:348] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.878211 10172 raft_consensus.cc:374] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.878322 10172 raft_consensus.cc:729] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.878248 10169 consensus_queue.cc:260] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.878610 10172 consensus_queue.cc:260] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.879098 10169 ts_tablet_manager.cc:1398] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:14:48.879453 10169 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:48.879670 10167 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:48.880007 10172 ts_tablet_manager.cc:1398] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:14:48.880273 10172 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:48.881325 10167 tablet_bootstrap.cc:654] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.881821 10172 tablet_bootstrap.cc:654] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.882743 10169 tablet_bootstrap.cc:654] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.885351 10101 raft_consensus.cc:482] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:48.885517 10101 raft_consensus.cc:504] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.886296 10167 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:48.886462 10167 ts_tablet_manager.cc:1367] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240503 02:14:48.886802 10101 leader_election.cc:288] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:48.887149 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:48.887277 10171 tablet_bootstrap.cc:654] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
W20240503 02:14:48.887713 8624 leader_election.cc:341] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:48.887713 10167 raft_consensus.cc:348] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.887826 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:48.888000 10167 raft_consensus.cc:374] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.888103 10167 raft_consensus.cc:729] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.888311 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:48.888495 10166 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:48.888684 10166 ts_tablet_manager.cc:1367] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.014s
I20240503 02:14:48.889643 10166 raft_consensus.cc:348] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
W20240503 02:14:48.889827 8625 leader_election.cc:341] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:48.889926 10166 raft_consensus.cc:374] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.890005 10166 raft_consensus.cc:729] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.889901 10167 consensus_queue.cc:260] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
W20240503 02:14:48.890331 8625 leader_election.cc:341] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:48.890292 10166 consensus_queue.cc:260] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.891409 10167 ts_tablet_manager.cc:1398] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:14:48.891355 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:48.891606 8821 raft_consensus.cc:2454] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 0.
I20240503 02:14:48.891798 10167 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:48.892060 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:48.893014 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:48.893280 8425 raft_consensus.cc:2454] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 0.
I20240503 02:14:48.893828 10167 tablet_bootstrap.cc:654] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.895432 10172 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:48.895639 10172 ts_tablet_manager.cc:1367] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
W20240503 02:14:48.895999 8623 leader_election.cc:341] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:48.896132 8623 leader_election.cc:302] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, 846e89dd0b154d2b9bb7d4f315c9fc12; no voters: 2a1893ab239f4163a06bdbf8ce579431, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d, cb90f34dc37b4b16b2a5b5a4ee012865
I20240503 02:14:48.896538 10172 raft_consensus.cc:348] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.896857 10172 raft_consensus.cc:374] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.896932 10172 raft_consensus.cc:729] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.897331 10166 ts_tablet_manager.cc:1398] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.009s user 0.007s sys 0.002s
I20240503 02:14:48.897200 10172 consensus_queue.cc:260] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.898336 10172 ts_tablet_manager.cc:1398] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:14:48.898645 10172 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:48.898661 10101 raft_consensus.cc:2726] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:48.899206 10166 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:48.900952 10172 tablet_bootstrap.cc:654] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.902598 10166 tablet_bootstrap.cc:654] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.903757 10173 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:48.905354 10169 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:48.905400 10173 ts_tablet_manager.cc:1367] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.001s
I20240503 02:14:48.905544 10169 ts_tablet_manager.cc:1367] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.003s
I20240503 02:14:48.906265 10169 raft_consensus.cc:348] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.906549 10167 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:48.906556 10169 raft_consensus.cc:374] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.906327 10173 raft_consensus.cc:348] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.906638 10169 raft_consensus.cc:729] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.906756 10167 ts_tablet_manager.cc:1367] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20240503 02:14:48.906781 10173 raft_consensus.cc:374] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.906927 10173 raft_consensus.cc:729] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.906925 10169 consensus_queue.cc:260] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.907692 10169 ts_tablet_manager.cc:1398] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:48.907536 10167 raft_consensus.cc:348] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.907922 10167 raft_consensus.cc:374] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.908017 10167 raft_consensus.cc:729] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.908334 10169 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:48.908715 10167 consensus_queue.cc:260] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.910044 10169 tablet_bootstrap.cc:654] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.910908 10167 ts_tablet_manager.cc:1398] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:14:48.911266 10167 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:48.907280 10173 consensus_queue.cc:260] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.914368 10173 ts_tablet_manager.cc:1398] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 02:14:48.916410 10173 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:48.916597 10172 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:48.916805 10172 ts_tablet_manager.cc:1367] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240503 02:14:48.917794 10172 raft_consensus.cc:348] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.918160 10172 raft_consensus.cc:374] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.918263 10172 raft_consensus.cc:729] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.918588 10172 consensus_queue.cc:260] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.920688 10172 ts_tablet_manager.cc:1398] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:14:48.921494 10167 tablet_bootstrap.cc:654] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.921552 10169 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:48.921705 10169 ts_tablet_manager.cc:1367] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240503 02:14:48.922485 10169 raft_consensus.cc:348] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.922835 10169 raft_consensus.cc:374] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.922967 10173 tablet_bootstrap.cc:654] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.922952 10169 raft_consensus.cc:729] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.923251 10169 consensus_queue.cc:260] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.924039 10169 ts_tablet_manager.cc:1398] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:14:48.926973 10168 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:48.927170 10168 ts_tablet_manager.cc:1367] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.000s
I20240503 02:14:48.927393 10101 raft_consensus.cc:482] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:48.927529 10101 raft_consensus.cc:504] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.927985 10168 raft_consensus.cc:348] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.928237 10168 raft_consensus.cc:374] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.928331 10168 raft_consensus.cc:729] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.928755 10101 leader_election.cc:288] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:48.929029 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:48.929342 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:48.929423 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
W20240503 02:14:48.929536 8624 leader_election.cc:341] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:48.929618 9213 raft_consensus.cc:2454] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 0.
I20240503 02:14:48.928608 10168 consensus_queue.cc:260] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.929920 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:48.930012 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:48.930282 8821 raft_consensus.cc:2454] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846e89dd0b154d2b9bb7d4f315c9fc12 in term 0.
W20240503 02:14:48.930478 8623 leader_election.cc:341] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:48.930449 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
W20240503 02:14:48.930946 8625 leader_election.cc:341] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:48.931186 10171 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:48.931335 10171 ts_tablet_manager.cc:1367] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.006s
W20240503 02:14:48.931416 8623 leader_election.cc:341] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:48.931526 8623 leader_election.cc:302] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 79e41f25066e4e5abbd71e48d545c6bf, 846e89dd0b154d2b9bb7d4f315c9fc12, cb90f34dc37b4b16b2a5b5a4ee012865; no voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d
I20240503 02:14:48.931823 10101 raft_consensus.cc:2726] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:48.932109 10171 raft_consensus.cc:348] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.932303 10171 raft_consensus.cc:374] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.932399 10171 raft_consensus.cc:729] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.932629 10171 consensus_queue.cc:260] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.933434 10167 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:48.933470 10168 ts_tablet_manager.cc:1398] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:14:48.933568 10167 ts_tablet_manager.cc:1367] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.002s
I20240503 02:14:48.933773 10168 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:48.934311 10167 raft_consensus.cc:348] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.934538 10167 raft_consensus.cc:374] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.934612 10167 raft_consensus.cc:729] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.934841 10167 consensus_queue.cc:260] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.935372 10168 tablet_bootstrap.cc:654] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.935972 10171 ts_tablet_manager.cc:1398] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:14:48.936286 10171 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:48.936570 10167 ts_tablet_manager.cc:1398] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:14:48.940030 10171 tablet_bootstrap.cc:654] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.940297 10173 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:48.940522 10173 ts_tablet_manager.cc:1367] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.004s
I20240503 02:14:48.941428 10173 raft_consensus.cc:348] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.942046 10173 raft_consensus.cc:374] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.942189 10173 raft_consensus.cc:729] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.942523 10173 consensus_queue.cc:260] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.943323 10173 ts_tablet_manager.cc:1398] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:14:48.943655 10173 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:48.944655 10166 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:48.944847 10166 ts_tablet_manager.cc:1367] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.011s
I20240503 02:14:48.945340 10173 tablet_bootstrap.cc:654] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.945520 10166 raft_consensus.cc:348] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.945799 10166 raft_consensus.cc:374] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.945880 10166 raft_consensus.cc:729] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.947695 10173 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:48.947829 10168 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:48.947858 10173 ts_tablet_manager.cc:1367] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:14:48.948005 10168 ts_tablet_manager.cc:1367] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 02:14:48.948519 10173 raft_consensus.cc:348] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.948743 10173 raft_consensus.cc:374] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.948856 10173 raft_consensus.cc:729] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.949082 10173 consensus_queue.cc:260] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.949694 10173 ts_tablet_manager.cc:1398] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:14:48.949978 10189 raft_consensus.cc:482] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:48.950150 10189 raft_consensus.cc:504] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.950229 10168 raft_consensus.cc:348] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.951648 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:48.951821 10168 raft_consensus.cc:374] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.951902 10168 raft_consensus.cc:729] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.951887 8425 raft_consensus.cc:2454] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:48.952507 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:48.952636 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:48.952847 9213 raft_consensus.cc:2454] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
W20240503 02:14:48.953045 8888 leader_election.cc:341] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:48.946141 10166 consensus_queue.cc:260] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.953395 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:48.953590 8821 raft_consensus.cc:2454] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:48.953601 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:48.953747 10166 ts_tablet_manager.cc:1398] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 02:14:48.953846 8689 raft_consensus.cc:2454] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:48.954056 10166 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:48.954260 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:48.954463 9082 raft_consensus.cc:2454] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:48.954337 10168 consensus_queue.cc:260] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.955256 10168 ts_tablet_manager.cc:1398] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:14:48.955610 10166 tablet_bootstrap.cc:654] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.955611 10168 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:48.956301 8886 leader_election.cc:302] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d, cb90f34dc37b4b16b2a5b5a4ee012865; no voters: 2a1893ab239f4163a06bdbf8ce579431
I20240503 02:14:48.957293 10168 tablet_bootstrap.cc:654] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.951483 10189 leader_election.cc:288] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:48.960067 10189 raft_consensus.cc:2781] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:48.960175 10189 raft_consensus.cc:482] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:48.960268 10189 raft_consensus.cc:3037] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:48.962245 10168 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:48.962405 10168 ts_tablet_manager.cc:1367] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 02:14:48.963119 10168 raft_consensus.cc:348] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.963364 10168 raft_consensus.cc:374] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.963464 10168 raft_consensus.cc:729] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.965143 10168 consensus_queue.cc:260] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.967145 10168 ts_tablet_manager.cc:1398] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:14:48.967295 10189 raft_consensus.cc:504] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.968640 10171 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:48.968674 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:48.968833 10171 ts_tablet_manager.cc:1367] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.003s
I20240503 02:14:48.968888 8425 raft_consensus.cc:3037] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:48.969642 10171 raft_consensus.cc:348] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.969939 10171 raft_consensus.cc:374] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.970016 10171 raft_consensus.cc:729] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.970281 10171 consensus_queue.cc:260] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:48.970984 10171 ts_tablet_manager.cc:1398] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 02:14:48.971866 10171 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:48.971873 8425 raft_consensus.cc:2454] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:48.972404 10166 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:48.972553 10166 ts_tablet_manager.cc:1367] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.012s
I20240503 02:14:48.973541 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:48.973706 10171 tablet_bootstrap.cc:654] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:48.973666 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:48.973889 8689 raft_consensus.cc:3037] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:14:48.974123 8888 leader_election.cc:341] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:48.973690 10166 raft_consensus.cc:348] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.974370 10166 raft_consensus.cc:374] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:48.974462 10166 raft_consensus.cc:729] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:48.974714 10166 consensus_queue.cc:260] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.975870 10166 ts_tablet_manager.cc:1398] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:14:48.976579 10160 raft_consensus.cc:482] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:48.976804 10160 raft_consensus.cc:504] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.978569 10160 leader_election.cc:288] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:48.978729 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bca5fd721f84f38bff08308965eb33f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:48.978870 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bca5fd721f84f38bff08308965eb33f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:48.978929 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bca5fd721f84f38bff08308965eb33f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:48.978958 9213 raft_consensus.cc:2454] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 0.
I20240503 02:14:48.979110 8949 raft_consensus.cc:2454] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 0.
I20240503 02:14:48.979111 8558 raft_consensus.cc:2454] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 0.
I20240503 02:14:48.979337 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bca5fd721f84f38bff08308965eb33f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:48.979559 8690 raft_consensus.cc:2454] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 0.
I20240503 02:14:48.979763 8756 leader_election.cc:302] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:48.980284 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bca5fd721f84f38bff08308965eb33f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:48.980468 10160 raft_consensus.cc:2781] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:48.980504 9082 raft_consensus.cc:2454] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 0.
I20240503 02:14:48.980597 10160 raft_consensus.cc:482] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:48.980676 10160 raft_consensus.cc:3037] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:48.981885 8689 raft_consensus.cc:2454] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:48.982708 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bca5fd721f84f38bff08308965eb33f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:48.982901 8425 raft_consensus.cc:2454] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 0.
I20240503 02:14:48.982995 10160 raft_consensus.cc:504] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.984330 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bca5fd721f84f38bff08308965eb33f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:48.984541 9213 raft_consensus.cc:3037] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:48.984509 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bca5fd721f84f38bff08308965eb33f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:48.984707 8949 raft_consensus.cc:3037] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:48.985159 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bca5fd721f84f38bff08308965eb33f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:48.985405 9082 raft_consensus.cc:3037] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:48.987331 9213 raft_consensus.cc:2454] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 1.
I20240503 02:14:48.987434 8949 raft_consensus.cc:2454] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 1.
I20240503 02:14:48.989063 10160 leader_election.cc:288] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 election: Requested vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:48.989288 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bca5fd721f84f38bff08308965eb33f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:48.989522 8558 raft_consensus.cc:3037] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:48.990401 9082 raft_consensus.cc:2454] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 1.
I20240503 02:14:48.991436 8558 raft_consensus.cc:2454] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 1.
I20240503 02:14:48.991943 8757 leader_election.cc:302] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:48.992504 10197 raft_consensus.cc:2781] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:48.992724 10197 raft_consensus.cc:686] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEADER]: Becoming Leader. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Running, Role: LEADER
I20240503 02:14:48.993108 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bca5fd721f84f38bff08308965eb33f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:48.993331 8689 raft_consensus.cc:3037] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:48.993220 10197 consensus_queue.cc:237] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:48.994196 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bca5fd721f84f38bff08308965eb33f" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:48.994400 8425 raft_consensus.cc:3037] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:48.996703 10189 leader_election.cc:288] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 election: Requested vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:48.996968 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:48.997188 8821 raft_consensus.cc:3037] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:48.997217 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:48.997504 9082 raft_consensus.cc:3037] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:48.998805 8689 raft_consensus.cc:2454] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 1.
I20240503 02:14:48.999945 8821 raft_consensus.cc:2454] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:48.999819 8273 catalog_manager.cc:5484] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196). New cstate: current_term: 1 leader_uuid: "79e41f25066e4e5abbd71e48d545c6bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:49.000391 8886 leader_election.cc:302] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, 846e89dd0b154d2b9bb7d4f315c9fc12, 99be4ff1114d4aaabcc537604fce54b3; no voters: 2a1893ab239f4163a06bdbf8ce579431
I20240503 02:14:49.000842 9082 raft_consensus.cc:2454] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:49.001030 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:49.001235 9213 raft_consensus.cc:3037] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:49.003624 9213 raft_consensus.cc:2454] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:49.003945 10189 raft_consensus.cc:2781] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:49.004120 10189 raft_consensus.cc:686] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Becoming Leader. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Running, Role: LEADER
I20240503 02:14:49.004616 10189 consensus_queue.cc:237] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:49.004797 10171 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:49.004931 10171 ts_tablet_manager.cc:1367] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.001s
I20240503 02:14:49.005484 10171 raft_consensus.cc:348] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:49.005700 10171 raft_consensus.cc:374] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:49.005798 10171 raft_consensus.cc:729] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:49.006022 10171 consensus_queue.cc:260] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:49.006625 8425 raft_consensus.cc:2454] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 1.
I20240503 02:14:49.006654 10171 ts_tablet_manager.cc:1398] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:14:49.009176 8273 catalog_manager.cc:5484] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197). New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:49.173120 10189 raft_consensus.cc:482] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:49.173362 10189 raft_consensus.cc:504] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:49.174520 10189 leader_election.cc:288] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:49.174507 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:49.174657 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:49.174769 8558 raft_consensus.cc:2454] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:49.174924 8425 raft_consensus.cc:2454] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:49.175096 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:49.175226 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:49.175329 9082 raft_consensus.cc:2454] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:49.175488 8689 raft_consensus.cc:2454] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:49.175549 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:49.175666 8886 leader_election.cc:302] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d; no voters:
I20240503 02:14:49.175762 8821 raft_consensus.cc:2454] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:49.175966 10189 raft_consensus.cc:2781] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:49.176045 10189 raft_consensus.cc:482] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:49.176023 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:49.176163 10189 raft_consensus.cc:3037] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:49.176286 9213 raft_consensus.cc:2454] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:49.178277 10189 raft_consensus.cc:504] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:49.179492 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:49.179587 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:49.179731 8558 raft_consensus.cc:3037] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:49.179836 8425 raft_consensus.cc:3037] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:49.179883 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:49.180091 8689 raft_consensus.cc:3037] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:49.180344 10189 leader_election.cc:288] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 election: Requested vote from peers 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:49.180656 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:49.180833 8821 raft_consensus.cc:3037] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:49.182580 8558 raft_consensus.cc:2454] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:49.182735 8425 raft_consensus.cc:2454] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:49.182832 8689 raft_consensus.cc:2454] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:49.183569 8821 raft_consensus.cc:2454] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:49.183590 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:49.183633 8888 leader_election.cc:302] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 846e89dd0b154d2b9bb7d4f315c9fc12, 99be4ff1114d4aaabcc537604fce54b3; no voters:
I20240503 02:14:49.183822 9082 raft_consensus.cc:3037] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:49.184062 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a8f2627c5c49a39291c96eb00fec18" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:49.184259 10189 raft_consensus.cc:2781] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:49.184286 9213 raft_consensus.cc:3037] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:49.184419 10189 raft_consensus.cc:686] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Becoming Leader. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Running, Role: LEADER
I20240503 02:14:49.184751 10189 consensus_queue.cc:237] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:49.186424 9213 raft_consensus.cc:2454] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:49.186466 9082 raft_consensus.cc:2454] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:49.189594 8278 catalog_manager.cc:5484] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197). New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:49.406623 10197 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.434984 10193 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.437455 8558 raft_consensus.cc:3037] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:49.441252 10193 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.447942 10197 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.450692 10193 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.453950 10197 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.458055 10193 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.459348 10193 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.462565 10193 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240503 02:14:49.489745 10160 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.495469 10160 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.523432 10197 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.589080 10209 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.594565 10209 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.637488 10209 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.644966 10189 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.645380 10193 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:49.645762 10189 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:49.680837 10203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:49.681133 10203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:49.688606 10203 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 428.452 ms
time per row: 0.214226 ms
I20240503 02:14:50.272714 8273 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42220:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:14:50.273406 8273 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_4 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 02:14:50.282281 8403 tablet_service.cc:1467] Processing CreateTablet for tablet 5028c92b346845c38b3356a0d8c7507f (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.282725 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5028c92b346845c38b3356a0d8c7507f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.283900 9193 tablet_service.cc:1467] Processing CreateTablet for tablet 5028c92b346845c38b3356a0d8c7507f (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.284309 9193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5028c92b346845c38b3356a0d8c7507f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.284219 8668 tablet_service.cc:1467] Processing CreateTablet for tablet 5028c92b346845c38b3356a0d8c7507f (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.284637 8668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5028c92b346845c38b3356a0d8c7507f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.285089 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 4679ab055c1a40f19bf7b602451044f6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.285465 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4679ab055c1a40f19bf7b602451044f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.288434 9063 tablet_service.cc:1467] Processing CreateTablet for tablet 5028c92b346845c38b3356a0d8c7507f (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.288861 9063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5028c92b346845c38b3356a0d8c7507f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.288836 8930 tablet_service.cc:1467] Processing CreateTablet for tablet 5028c92b346845c38b3356a0d8c7507f (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.289261 8930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5028c92b346845c38b3356a0d8c7507f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.292218 8931 tablet_service.cc:1467] Processing CreateTablet for tablet 4679ab055c1a40f19bf7b602451044f6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.292590 8931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4679ab055c1a40f19bf7b602451044f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.292487 8405 tablet_service.cc:1467] Processing CreateTablet for tablet 4679ab055c1a40f19bf7b602451044f6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.293010 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4679ab055c1a40f19bf7b602451044f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.293270 8669 tablet_service.cc:1467] Processing CreateTablet for tablet 4679ab055c1a40f19bf7b602451044f6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.293654 8669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4679ab055c1a40f19bf7b602451044f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.296154 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 4679ab055c1a40f19bf7b602451044f6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.296562 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4679ab055c1a40f19bf7b602451044f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.300235 8800 tablet_service.cc:1467] Processing CreateTablet for tablet 5028c92b346845c38b3356a0d8c7507f (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.300665 8800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5028c92b346845c38b3356a0d8c7507f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.300609 8404 tablet_service.cc:1467] Processing CreateTablet for tablet a7a434e2df0a4ea0b2ab5c20034033c6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.300992 8404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a434e2df0a4ea0b2ab5c20034033c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.304140 8799 tablet_service.cc:1467] Processing CreateTablet for tablet 4679ab055c1a40f19bf7b602451044f6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.305397 8538 tablet_service.cc:1467] Processing CreateTablet for tablet 5028c92b346845c38b3356a0d8c7507f (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.305509 8537 tablet_service.cc:1467] Processing CreateTablet for tablet 4679ab055c1a40f19bf7b602451044f6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.305821 8538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5028c92b346845c38b3356a0d8c7507f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.306180 8537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4679ab055c1a40f19bf7b602451044f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.314890 8536 tablet_service.cc:1467] Processing CreateTablet for tablet a7a434e2df0a4ea0b2ab5c20034033c6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.315287 8536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a434e2df0a4ea0b2ab5c20034033c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.318225 10309 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:50.319197 8799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4679ab055c1a40f19bf7b602451044f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.322065 10309 tablet_bootstrap.cc:654] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.323685 10308 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:50.324313 8801 tablet_service.cc:1467] Processing CreateTablet for tablet a7a434e2df0a4ea0b2ab5c20034033c6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.324757 8801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a434e2df0a4ea0b2ab5c20034033c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.324779 10310 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:50.326704 10308 tablet_bootstrap.cc:654] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.332028 8668 tablet_service.cc:1467] Processing CreateTablet for tablet a7a434e2df0a4ea0b2ab5c20034033c6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.332674 8668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a434e2df0a4ea0b2ab5c20034033c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.332850 9063 tablet_service.cc:1467] Processing CreateTablet for tablet a7a434e2df0a4ea0b2ab5c20034033c6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.333065 10310 tablet_bootstrap.cc:654] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.333356 9063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a434e2df0a4ea0b2ab5c20034033c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.333693 9192 tablet_service.cc:1467] Processing CreateTablet for tablet a7a434e2df0a4ea0b2ab5c20034033c6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.334154 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a434e2df0a4ea0b2ab5c20034033c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.342483 8931 tablet_service.cc:1467] Processing CreateTablet for tablet a7a434e2df0a4ea0b2ab5c20034033c6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:50.342898 8931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a434e2df0a4ea0b2ab5c20034033c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:50.346778 10310 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:50.347424 10311 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:50.348172 10310 ts_tablet_manager.cc:1367] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.004s
I20240503 02:14:50.349141 10311 tablet_bootstrap.cc:654] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.351140 10310 raft_consensus.cc:348] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.352808 10310 raft_consensus.cc:374] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.353168 10310 raft_consensus.cc:729] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.353490 10308 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:50.353864 10310 consensus_queue.cc:260] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.355435 10308 ts_tablet_manager.cc:1367] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.004s
I20240503 02:14:50.355655 10314 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:50.356868 10310 ts_tablet_manager.cc:1398] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:14:50.357240 10314 tablet_bootstrap.cc:654] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.356379 10308 raft_consensus.cc:348] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.357331 10308 raft_consensus.cc:374] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.357895 10310 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:50.357940 10308 raft_consensus.cc:729] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.358611 10312 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:50.358629 10308 consensus_queue.cc:260] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.360414 10312 tablet_bootstrap.cc:654] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.360571 10310 tablet_bootstrap.cc:654] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.361857 10306 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:50.362069 10311 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:50.362226 10311 ts_tablet_manager.cc:1367] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20240503 02:14:50.362728 10309 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:50.362929 10309 ts_tablet_manager.cc:1367] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.008s
I20240503 02:14:50.363023 10311 raft_consensus.cc:348] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.363320 10311 raft_consensus.cc:374] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.363442 10311 raft_consensus.cc:729] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.363727 10311 consensus_queue.cc:260] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.363979 10309 raft_consensus.cc:348] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.363509 10306 tablet_bootstrap.cc:654] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.364347 10309 raft_consensus.cc:374] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.364459 10309 raft_consensus.cc:729] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.364766 10309 consensus_queue.cc:260] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.365106 10311 ts_tablet_manager.cc:1398] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:14:50.365427 10311 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:50.366503 10314 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:50.366704 10314 ts_tablet_manager.cc:1367] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 02:14:50.367170 10311 tablet_bootstrap.cc:654] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.368860 10314 raft_consensus.cc:348] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.369283 10314 raft_consensus.cc:374] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.369428 10314 raft_consensus.cc:729] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.369879 10312 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:50.370013 10312 ts_tablet_manager.cc:1367] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 02:14:50.369760 10314 consensus_queue.cc:260] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.370355 10308 ts_tablet_manager.cc:1398] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:14:50.370765 10308 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:50.370715 10312 raft_consensus.cc:348] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.370995 10312 raft_consensus.cc:374] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.371083 10312 raft_consensus.cc:729] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.371341 10312 consensus_queue.cc:260] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.372776 10308 tablet_bootstrap.cc:654] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.373768 10312 ts_tablet_manager.cc:1398] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:14:50.374104 10312 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:50.376559 10312 tablet_bootstrap.cc:654] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.378232 10314 ts_tablet_manager.cc:1398] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:14:50.379357 10314 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:50.379868 10311 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:50.380606 10311 ts_tablet_manager.cc:1367] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20240503 02:14:50.381695 10311 raft_consensus.cc:348] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.382045 10306 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:50.382277 10306 ts_tablet_manager.cc:1367] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20240503 02:14:50.383071 10311 raft_consensus.cc:374] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.383123 10314 tablet_bootstrap.cc:654] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.383316 10311 raft_consensus.cc:729] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.383390 10306 raft_consensus.cc:348] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.383742 10306 raft_consensus.cc:374] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.383589 10311 consensus_queue.cc:260] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.383864 10306 raft_consensus.cc:729] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.384188 10306 consensus_queue.cc:260] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.384944 10311 ts_tablet_manager.cc:1398] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:14:50.385064 10309 ts_tablet_manager.cc:1398] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20240503 02:14:50.385217 10311 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:50.388435 10311 tablet_bootstrap.cc:654] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.388973 10306 ts_tablet_manager.cc:1398] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:14:50.389837 10306 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:50.391880 10310 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:50.391978 10306 tablet_bootstrap.cc:654] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.392246 10310 ts_tablet_manager.cc:1367] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.011s
I20240503 02:14:50.393051 10310 raft_consensus.cc:348] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.393409 10310 raft_consensus.cc:374] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.393533 10310 raft_consensus.cc:729] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.393862 10310 consensus_queue.cc:260] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.394398 10309 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:50.394685 10310 ts_tablet_manager.cc:1398] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:50.394879 10312 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:50.395078 10310 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:50.395097 10312 ts_tablet_manager.cc:1367] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.005s
I20240503 02:14:50.396409 10312 raft_consensus.cc:348] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.396724 10312 raft_consensus.cc:374] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.396757 10310 tablet_bootstrap.cc:654] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.396842 10312 raft_consensus.cc:729] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.397102 10312 consensus_queue.cc:260] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.398636 10312 ts_tablet_manager.cc:1398] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:14:50.398990 10312 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:50.399048 10306 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:50.399150 10309 tablet_bootstrap.cc:654] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.399271 10306 ts_tablet_manager.cc:1367] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240503 02:14:50.400089 10306 raft_consensus.cc:348] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.400426 10306 raft_consensus.cc:374] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.400537 10306 raft_consensus.cc:729] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.400769 10308 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:50.400842 10306 consensus_queue.cc:260] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.401022 10308 ts_tablet_manager.cc:1367] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240503 02:14:50.401558 10306 ts_tablet_manager.cc:1398] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:50.401558 10311 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:50.401705 10311 ts_tablet_manager.cc:1367] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240503 02:14:50.401906 10306 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:50.402453 10308 raft_consensus.cc:348] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.402478 10311 raft_consensus.cc:348] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.402736 10311 raft_consensus.cc:374] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.402858 10311 raft_consensus.cc:729] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.404057 10312 tablet_bootstrap.cc:654] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.404771 10308 raft_consensus.cc:374] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.404979 10306 tablet_bootstrap.cc:654] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.404878 10311 consensus_queue.cc:260] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.405200 10308 raft_consensus.cc:729] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.405537 10311 ts_tablet_manager.cc:1398] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:14:50.405942 10308 consensus_queue.cc:260] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.406807 10312 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:50.407004 10312 ts_tablet_manager.cc:1367] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 02:14:50.407588 10310 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:50.407742 10310 ts_tablet_manager.cc:1367] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240503 02:14:50.408084 10308 ts_tablet_manager.cc:1398] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:14:50.408567 10310 raft_consensus.cc:348] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.408891 10310 raft_consensus.cc:374] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.409030 10310 raft_consensus.cc:729] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.409313 10310 consensus_queue.cc:260] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.409544 10309 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:50.409736 10309 ts_tablet_manager.cc:1367] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.005s
I20240503 02:14:50.410176 10310 ts_tablet_manager.cc:1398] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:14:50.411656 10306 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:50.411871 10306 ts_tablet_manager.cc:1367] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 02:14:50.412593 10306 raft_consensus.cc:348] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.412674 10309 raft_consensus.cc:348] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.412954 10306 raft_consensus.cc:374] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.413095 10306 raft_consensus.cc:729] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.413103 10312 raft_consensus.cc:348] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.413563 10312 raft_consensus.cc:374] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.413383 10306 consensus_queue.cc:260] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.413796 10308 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:50.414085 10306 ts_tablet_manager.cc:1398] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:50.413748 10312 raft_consensus.cc:729] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.414711 10309 raft_consensus.cc:374] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.414803 10309 raft_consensus.cc:729] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.415036 10309 consensus_queue.cc:260] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.415027 10312 consensus_queue.cc:260] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.415974 10308 tablet_bootstrap.cc:654] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.416453 10312 ts_tablet_manager.cc:1398] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 02:14:50.418510 10309 ts_tablet_manager.cc:1398] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240503 02:14:50.418782 10314 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:50.418797 10309 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:50.419041 10314 ts_tablet_manager.cc:1367] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.004s
I20240503 02:14:50.419991 10309 tablet_bootstrap.cc:654] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.419929 10314 raft_consensus.cc:348] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.420241 10314 raft_consensus.cc:374] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.420365 10314 raft_consensus.cc:729] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.420687 10314 consensus_queue.cc:260] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.421664 10314 ts_tablet_manager.cc:1398] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:14:50.421716 10309 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:50.421864 10309 ts_tablet_manager.cc:1367] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:14:50.422402 10314 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:50.422598 10309 raft_consensus.cc:348] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.422808 10309 raft_consensus.cc:374] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.422912 10309 raft_consensus.cc:729] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.423138 10309 consensus_queue.cc:260] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.423970 10308 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:50.424001 10314 tablet_bootstrap.cc:654] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:50.424111 10308 ts_tablet_manager.cc:1367] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20240503 02:14:50.424767 10308 raft_consensus.cc:348] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.425151 10308 raft_consensus.cc:374] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.425292 10308 raft_consensus.cc:729] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.425556 10308 consensus_queue.cc:260] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.426100 10308 ts_tablet_manager.cc:1398] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:14:50.426818 10309 ts_tablet_manager.cc:1398] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:14:50.429781 10314 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:50.429922 10314 ts_tablet_manager.cc:1367] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240503 02:14:50.430588 10314 raft_consensus.cc:348] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.430841 10314 raft_consensus.cc:374] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:50.430996 10314 raft_consensus.cc:729] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:50.431283 10314 consensus_queue.cc:260] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.431954 10314 ts_tablet_manager.cc:1398] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:14:50.437795 10150 raft_consensus.cc:482] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:50.437919 10150 raft_consensus.cc:504] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.438880 10150 leader_election.cc:288] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:50.438992 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c92b346845c38b3356a0d8c7507f" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:50.439155 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c92b346845c38b3356a0d8c7507f" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:50.439201 8558 raft_consensus.cc:2454] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:50.439306 8423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c92b346845c38b3356a0d8c7507f" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:50.439347 8689 raft_consensus.cc:2454] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:50.439491 8423 raft_consensus.cc:2454] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:50.439708 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c92b346845c38b3356a0d8c7507f" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:50.439976 9214 raft_consensus.cc:2454] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:50.439998 9017 leader_election.cc:302] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 846e89dd0b154d2b9bb7d4f315c9fc12, c399efcbba924863ac89d156dc07ac7d; no voters:
I20240503 02:14:50.440419 10150 raft_consensus.cc:2781] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:50.440377 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c92b346845c38b3356a0d8c7507f" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:50.440549 10150 raft_consensus.cc:482] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:50.440657 10150 raft_consensus.cc:3037] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.440634 8821 raft_consensus.cc:2454] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:50.440665 10209 raft_consensus.cc:482] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:50.440857 10209 raft_consensus.cc:504] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.441342 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c92b346845c38b3356a0d8c7507f" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:50.441637 8949 raft_consensus.cc:2454] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:50.442013 10209 leader_election.cc:288] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:50.442432 10307 raft_consensus.cc:482] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:50.442603 10307 raft_consensus.cc:504] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.443157 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a434e2df0a4ea0b2ab5c20034033c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:50.443303 8821 raft_consensus.cc:2454] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:50.443284 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a434e2df0a4ea0b2ab5c20034033c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:50.443460 8689 raft_consensus.cc:2454] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:50.443878 10307 leader_election.cc:288] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:50.443869 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a434e2df0a4ea0b2ab5c20034033c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:50.444082 9214 raft_consensus.cc:2454] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:50.442611 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a434e2df0a4ea0b2ab5c20034033c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:50.444626 8558 raft_consensus.cc:2454] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:50.444710 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4679ab055c1a40f19bf7b602451044f6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:50.444902 8821 raft_consensus.cc:2454] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:50.445394 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4679ab055c1a40f19bf7b602451044f6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:50.445427 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4679ab055c1a40f19bf7b602451044f6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:50.445469 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4679ab055c1a40f19bf7b602451044f6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:50.445566 8690 raft_consensus.cc:2454] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:50.445644 9214 raft_consensus.cc:2454] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:50.445662 8949 raft_consensus.cc:2454] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:50.446007 8493 leader_election.cc:302] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 79e41f25066e4e5abbd71e48d545c6bf, 846e89dd0b154d2b9bb7d4f315c9fc12, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:50.446270 10307 raft_consensus.cc:2781] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:50.446413 10307 raft_consensus.cc:482] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:50.446521 10307 raft_consensus.cc:3037] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.446767 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a434e2df0a4ea0b2ab5c20034033c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:50.447002 9083 raft_consensus.cc:2454] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:50.447522 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4679ab055c1a40f19bf7b602451044f6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:50.447814 9082 raft_consensus.cc:2454] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:50.448545 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a434e2df0a4ea0b2ab5c20034033c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:50.448747 8425 raft_consensus.cc:2454] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:50.449199 8886 leader_election.cc:302] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:50.449491 10209 raft_consensus.cc:2781] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:50.449587 10209 raft_consensus.cc:482] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:50.449677 10209 raft_consensus.cc:3037] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.450103 8423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4679ab055c1a40f19bf7b602451044f6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:50.450286 8423 raft_consensus.cc:2454] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:50.450524 10307 raft_consensus.cc:504] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.451637 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4679ab055c1a40f19bf7b602451044f6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:50.451807 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4679ab055c1a40f19bf7b602451044f6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:50.451857 8821 raft_consensus.cc:3037] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.452013 8949 raft_consensus.cc:3037] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.452297 10209 raft_consensus.cc:504] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.454181 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a434e2df0a4ea0b2ab5c20034033c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:50.454366 9214 raft_consensus.cc:3037] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.454911 8821 raft_consensus.cc:2454] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:50.454952 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a434e2df0a4ea0b2ab5c20034033c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:50.455128 8558 raft_consensus.cc:3037] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.456110 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a434e2df0a4ea0b2ab5c20034033c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:50.456264 8690 raft_consensus.cc:3037] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.457203 9214 raft_consensus.cc:2454] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:50.457623 8949 raft_consensus.cc:2454] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:50.457926 10209 leader_election.cc:288] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 election: Requested vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:50.458403 8690 raft_consensus.cc:2454] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:50.458722 8558 raft_consensus.cc:2454] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:50.458978 8423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a434e2df0a4ea0b2ab5c20034033c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:50.459175 8423 raft_consensus.cc:3037] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.459142 8888 leader_election.cc:302] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 846e89dd0b154d2b9bb7d4f315c9fc12, 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:50.459164 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a434e2df0a4ea0b2ab5c20034033c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:50.459251 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4679ab055c1a40f19bf7b602451044f6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:50.459421 9083 raft_consensus.cc:3037] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.459447 9082 raft_consensus.cc:3037] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.459756 10307 leader_election.cc:288] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 election: Requested vote from peers 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:50.461611 8423 raft_consensus.cc:2454] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:50.462054 10209 raft_consensus.cc:2781] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:50.462251 10209 raft_consensus.cc:686] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Becoming Leader. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Running, Role: LEADER
I20240503 02:14:50.462366 9083 raft_consensus.cc:2454] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:50.462576 10209 consensus_queue.cc:237] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.463294 8491 leader_election.cc:302] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d; no voters:
I20240503 02:14:50.463393 8423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4679ab055c1a40f19bf7b602451044f6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:50.463575 10307 raft_consensus.cc:2781] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:50.463639 8423 raft_consensus.cc:3037] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.464123 10307 raft_consensus.cc:686] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Becoming Leader. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Running, Role: LEADER
I20240503 02:14:50.464156 8821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a434e2df0a4ea0b2ab5c20034033c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:50.464401 8821 raft_consensus.cc:3037] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.464537 10307 consensus_queue.cc:237] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:50.464918 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4679ab055c1a40f19bf7b602451044f6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:50.465111 8690 raft_consensus.cc:3037] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.465402 10150 raft_consensus.cc:504] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.466820 8423 raft_consensus.cc:2454] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:50.466917 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c92b346845c38b3356a0d8c7507f" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:50.467082 8425 raft_consensus.cc:3037] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.467319 9082 raft_consensus.cc:2454] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:50.468504 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4679ab055c1a40f19bf7b602451044f6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:50.468683 9214 raft_consensus.cc:3037] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.469892 8425 raft_consensus.cc:2454] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:50.470523 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c92b346845c38b3356a0d8c7507f" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:50.470800 8558 raft_consensus.cc:3037] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.471415 9214 raft_consensus.cc:2454] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:50.473641 8558 raft_consensus.cc:2454] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:50.474174 8690 raft_consensus.cc:2454] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:50.476593 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c92b346845c38b3356a0d8c7507f" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:50.476794 8949 raft_consensus.cc:3037] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.476778 8820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c92b346845c38b3356a0d8c7507f" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:50.476989 8820 raft_consensus.cc:3037] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.477579 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c92b346845c38b3356a0d8c7507f" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:50.477763 8690 raft_consensus.cc:3037] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.477348 8282 catalog_manager.cc:5484] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197). New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:50.478098 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5028c92b346845c38b3356a0d8c7507f" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:50.478271 9214 raft_consensus.cc:3037] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:50.479430 8949 raft_consensus.cc:2454] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:50.479341 8273 catalog_manager.cc:5484] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:50.480509 8690 raft_consensus.cc:2454] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:50.481356 9018 leader_election.cc:302] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 4c668b2e2d5c4edba46cbc5c36065c0c, 846e89dd0b154d2b9bb7d4f315c9fc12, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d; no voters:
I20240503 02:14:50.481719 10336 raft_consensus.cc:2781] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:50.481997 9214 raft_consensus.cc:2454] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:50.482457 10150 leader_election.cc:288] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 election: Requested vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:50.484016 8821 raft_consensus.cc:2454] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:50.484602 8820 raft_consensus.cc:2454] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:50.491991 10336 raft_consensus.cc:686] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Becoming Leader. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Running, Role: LEADER
I20240503 02:14:50.492307 10336 consensus_queue.cc:237] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:50.495604 8273 catalog_manager.cc:5484] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c399efcbba924863ac89d156dc07ac7d (127.23.104.198). New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:50.859546 10209 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:50.874944 10339 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:50.882656 10209 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:50.885111 10341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:50.885398 10341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:50.892702 10341 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:14:50.922116 10209 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:50.923553 10307 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240503 02:14:50.929462 10212 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:50.929837 10212 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:50.939950 10307 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:50.949461 10336 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:50.960932 10212 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:50.970414 10339 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:50.997954 10307 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:51.020735 10307 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:51.051986 10307 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:51.053040 10336 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:51.061827 10307 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:51.067606 10339 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:51.204018 9017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.196:34747, user_credentials={real_user=slave}} blocked reactor thread for 136968us
I20240503 02:14:51.205638 10336 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:51.210460 9019 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:45923, user_credentials={real_user=slave}} blocked reactor thread for 84910.2us
W20240503 02:14:51.490499 8886 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:41237, user_credentials={real_user=slave}} blocked reactor thread for 64558.9us
INSERT report
rows total: 2000
time total: 469.426 ms
time per row: 0.234713 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:14:52.838614 8278 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42220:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:14:52.839320 8278 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_5 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 02:14:52.847422 8800 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea7444883c642419cd3e509f2b8d090 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.848294 9193 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea7444883c642419cd3e509f2b8d090 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.848750 9193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea7444883c642419cd3e509f2b8d090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.848688 8537 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea7444883c642419cd3e509f2b8d090 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.849061 8537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea7444883c642419cd3e509f2b8d090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.849675 9063 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea7444883c642419cd3e509f2b8d090 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.850112 9063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea7444883c642419cd3e509f2b8d090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.850214 8931 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea7444883c642419cd3e509f2b8d090 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.850621 8931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea7444883c642419cd3e509f2b8d090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.851594 8668 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea7444883c642419cd3e509f2b8d090 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.852017 8668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea7444883c642419cd3e509f2b8d090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.854807 8930 tablet_service.cc:1467] Processing CreateTablet for tablet 00fc93702514499eb32902238c46356d (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.854960 8932 tablet_service.cc:1467] Processing CreateTablet for tablet ac90f5b4e0fc43e99b3f76fe92610153 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.855181 8930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fc93702514499eb32902238c46356d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.855445 8932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac90f5b4e0fc43e99b3f76fe92610153. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.855911 8403 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea7444883c642419cd3e509f2b8d090 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.856312 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea7444883c642419cd3e509f2b8d090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.860291 10453 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:52.860790 8800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea7444883c642419cd3e509f2b8d090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.860741 8404 tablet_service.cc:1467] Processing CreateTablet for tablet 00fc93702514499eb32902238c46356d (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.861120 8404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fc93702514499eb32902238c46356d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.862030 10453 tablet_bootstrap.cc:654] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:52.863978 8801 tablet_service.cc:1467] Processing CreateTablet for tablet ac90f5b4e0fc43e99b3f76fe92610153 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.864354 8801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac90f5b4e0fc43e99b3f76fe92610153. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.865200 8537 tablet_service.cc:1467] Processing CreateTablet for tablet 00fc93702514499eb32902238c46356d (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.865563 8537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fc93702514499eb32902238c46356d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.865670 9194 tablet_service.cc:1467] Processing CreateTablet for tablet ac90f5b4e0fc43e99b3f76fe92610153 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.866016 9194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac90f5b4e0fc43e99b3f76fe92610153. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.868587 8538 tablet_service.cc:1467] Processing CreateTablet for tablet ac90f5b4e0fc43e99b3f76fe92610153 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.868976 8538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac90f5b4e0fc43e99b3f76fe92610153. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.872133 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 00fc93702514499eb32902238c46356d (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.872495 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fc93702514499eb32902238c46356d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.875850 10459 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:52.878651 10459 tablet_bootstrap.cc:654] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:52.880481 9062 tablet_service.cc:1467] Processing CreateTablet for tablet ac90f5b4e0fc43e99b3f76fe92610153 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.880916 8669 tablet_service.cc:1467] Processing CreateTablet for tablet 00fc93702514499eb32902238c46356d (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.880892 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 00fc93702514499eb32902238c46356d (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.881033 9062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac90f5b4e0fc43e99b3f76fe92610153. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.881294 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fc93702514499eb32902238c46356d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.881294 8669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fc93702514499eb32902238c46356d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.884163 8670 tablet_service.cc:1467] Processing CreateTablet for tablet ac90f5b4e0fc43e99b3f76fe92610153 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.884560 8670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac90f5b4e0fc43e99b3f76fe92610153. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.863925 8799 tablet_service.cc:1467] Processing CreateTablet for tablet 00fc93702514499eb32902238c46356d (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.889626 8799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fc93702514499eb32902238c46356d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.896122 10462 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:52.900121 10462 tablet_bootstrap.cc:654] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:52.904145 10460 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:52.905402 8404 tablet_service.cc:1467] Processing CreateTablet for tablet ac90f5b4e0fc43e99b3f76fe92610153 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:52.905686 10462 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:52.905835 10462 ts_tablet_manager.cc:1367] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240503 02:14:52.905848 8404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac90f5b4e0fc43e99b3f76fe92610153. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:52.906544 10462 raft_consensus.cc:348] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:52.906817 10462 raft_consensus.cc:374] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:52.906908 10462 raft_consensus.cc:729] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:52.907078 10460 tablet_bootstrap.cc:654] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:52.910863 10462 consensus_queue.cc:260] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:52.911479 10462 ts_tablet_manager.cc:1398] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:14:52.913944 10462 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:52.915860 10460 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:52.916663 10460 ts_tablet_manager.cc:1367] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240503 02:14:52.917686 10460 raft_consensus.cc:348] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:52.918998 10460 raft_consensus.cc:374] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:52.919135 10462 tablet_bootstrap.cc:654] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:52.919360 10460 raft_consensus.cc:729] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:52.919880 10460 consensus_queue.cc:260] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:52.921506 10460 ts_tablet_manager.cc:1398] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:14:52.922238 10460 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:52.924093 10460 tablet_bootstrap.cc:654] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:52.928663 10454 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:52.930613 10462 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:52.930830 10462 ts_tablet_manager.cc:1367] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20240503 02:14:52.931705 10462 raft_consensus.cc:348] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:52.932018 10462 raft_consensus.cc:374] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:52.932111 10462 raft_consensus.cc:729] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:52.932344 10454 tablet_bootstrap.cc:654] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:52.933027 10460 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:52.933737 10460 ts_tablet_manager.cc:1367] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240503 02:14:52.934993 10460 raft_consensus.cc:348] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:52.937310 10462 consensus_queue.cc:260] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:52.938680 10460 raft_consensus.cc:374] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:52.939016 10460 raft_consensus.cc:729] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:52.939658 10460 consensus_queue.cc:260] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:52.941036 10462 ts_tablet_manager.cc:1398] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20240503 02:14:52.941327 10462 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:52.942909 10462 tablet_bootstrap.cc:654] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:52.946382 10460 ts_tablet_manager.cc:1398] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:14:52.946990 10460 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:52.949404 10460 tablet_bootstrap.cc:654] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:52.954021 10454 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:52.954277 10454 ts_tablet_manager.cc:1367] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.011s
I20240503 02:14:52.955263 10454 raft_consensus.cc:348] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:52.955658 10454 raft_consensus.cc:374] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:52.955822 10454 raft_consensus.cc:729] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:52.956168 10460 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:52.956300 10460 ts_tablet_manager.cc:1367] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20240503 02:14:52.956101 10454 consensus_queue.cc:260] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:52.958369 10454 ts_tablet_manager.cc:1398] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:14:52.958259 10460 raft_consensus.cc:348] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:52.958562 10460 raft_consensus.cc:374] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:52.958670 10460 raft_consensus.cc:729] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:52.959113 10454 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:52.959510 10470 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:52.959631 10460 consensus_queue.cc:260] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:52.961794 10460 ts_tablet_manager.cc:1398] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:14:52.966650 10454 tablet_bootstrap.cc:654] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:52.963006 10470 tablet_bootstrap.cc:654] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
W20240503 02:14:52.973022 9148 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.193:38635, user_credentials={real_user=slave}} blocked reactor thread for 112088us
I20240503 02:14:52.973695 10454 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:52.973971 10454 ts_tablet_manager.cc:1367] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.001s
I20240503 02:14:52.974225 10457 raft_consensus.cc:482] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:52.974390 10457 raft_consensus.cc:504] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:52.974984 10454 raft_consensus.cc:348] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:52.975306 10454 raft_consensus.cc:374] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:52.975445 10454 raft_consensus.cc:729] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:52.975605 10457 leader_election.cc:288] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:52.975862 10454 consensus_queue.cc:260] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:52.976361 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:52.976574 10454 ts_tablet_manager.cc:1398] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:52.976965 10454 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:52.977252 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:52.977434 9082 raft_consensus.cc:2454] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 0.
I20240503 02:14:52.977494 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:52.977769 8950 raft_consensus.cc:2454] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e41f25066e4e5abbd71e48d545c6bf in term 0.
I20240503 02:14:52.977984 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
W20240503 02:14:52.978091 8755 leader_election.cc:341] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:14:52.978623 8757 leader_election.cc:341] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:52.979485 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
W20240503 02:14:52.980224 8755 leader_election.cc:341] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:52.980664 10454 tablet_bootstrap.cc:654] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:52.989681 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "79e41f25066e4e5abbd71e48d545c6bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
W20240503 02:14:52.990851 8757 leader_election.cc:341] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:52.991022 8757 leader_election.cc:302] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d; no voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 846e89dd0b154d2b9bb7d4f315c9fc12, cb90f34dc37b4b16b2a5b5a4ee012865
I20240503 02:14:52.991582 10457 raft_consensus.cc:2726] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:52.994218 10458 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:52.994381 10454 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:52.994540 10454 ts_tablet_manager.cc:1367] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20240503 02:14:52.995293 10454 raft_consensus.cc:348] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:52.995536 10454 raft_consensus.cc:374] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:52.995631 10454 raft_consensus.cc:729] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:52.997506 10454 consensus_queue.cc:260] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:52.998705 10454 ts_tablet_manager.cc:1398] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:14:53.009099 10440 raft_consensus.cc:482] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:53.009285 10440 raft_consensus.cc:504] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:53.010727 10440 leader_election.cc:288] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:53.014537 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:53.014822 8819 raft_consensus.cc:2454] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:53.015892 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:53.016116 9082 raft_consensus.cc:2454] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:53.016698 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:53.017858 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
W20240503 02:14:53.018990 8886 leader_election.cc:341] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:14:53.019379 8888 leader_election.cc:341] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:53.019804 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
W20240503 02:14:53.020458 8888 leader_election.cc:341] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:53.014325 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:53.034332 10458 tablet_bootstrap.cc:654] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:53.039083 10462 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:53.039271 10462 ts_tablet_manager.cc:1367] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.098s user 0.008s sys 0.019s
I20240503 02:14:53.040256 10462 raft_consensus.cc:348] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.040524 10462 raft_consensus.cc:374] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:53.040624 10462 raft_consensus.cc:729] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
W20240503 02:14:53.040685 8886 leader_election.cc:341] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:53.040814 8886 leader_election.cc:302] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d; no voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 846e89dd0b154d2b9bb7d4f315c9fc12, cb90f34dc37b4b16b2a5b5a4ee012865
I20240503 02:14:53.040928 10470 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:53.041083 10470 ts_tablet_manager.cc:1367] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.082s user 0.013s sys 0.033s
I20240503 02:14:53.041591 10423 raft_consensus.cc:2726] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:53.041656 10459 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:53.041803 10459 ts_tablet_manager.cc:1367] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.166s user 0.010s sys 0.036s
I20240503 02:14:53.041798 10470 raft_consensus.cc:348] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.042085 10470 raft_consensus.cc:374] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:53.042176 10470 raft_consensus.cc:729] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:53.042400 10470 consensus_queue.cc:260] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.042546 10382 raft_consensus.cc:482] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:53.042613 10459 raft_consensus.cc:348] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.042834 10459 raft_consensus.cc:374] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:53.042699 10382 raft_consensus.cc:504] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.042909 10459 raft_consensus.cc:729] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:53.042958 10470 ts_tablet_manager.cc:1398] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:14:53.043282 10470 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:53.043979 10462 consensus_queue.cc:260] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.044448 10459 consensus_queue.cc:260] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.044948 10470 tablet_bootstrap.cc:654] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:53.044988 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:53.045177 8819 raft_consensus.cc:2454] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:53.045512 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:53.045794 10462 ts_tablet_manager.cc:1398] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:14:53.046424 10459 ts_tablet_manager.cc:1398] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240503 02:14:53.046749 10459 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
W20240503 02:14:53.047427 9017 leader_election.cc:341] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:53.048243 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:53.048463 8950 raft_consensus.cc:2454] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:53.048882 10453 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:53.049036 10453 ts_tablet_manager.cc:1367] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.190s user 0.021s sys 0.039s
I20240503 02:14:53.049075 10459 tablet_bootstrap.cc:654] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:53.049865 10453 raft_consensus.cc:348] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.049964 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:53.050163 10453 raft_consensus.cc:374] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:53.050256 10453 raft_consensus.cc:729] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:53.050482 10453 consensus_queue.cc:260] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.051793 10453 ts_tablet_manager.cc:1398] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 02:14:53.052103 10453 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:53.053858 10453 tablet_bootstrap.cc:654] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
W20240503 02:14:53.056423 9019 leader_election.cc:341] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:53.056716 10458 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:53.056872 10458 ts_tablet_manager.cc:1367] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.063s user 0.000s sys 0.004s
I20240503 02:14:53.057340 10382 leader_election.cc:288] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:53.057572 10458 raft_consensus.cc:348] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.057819 10458 raft_consensus.cc:374] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:53.057895 10458 raft_consensus.cc:729] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:53.057932 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:53.058197 10458 consensus_queue.cc:260] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
W20240503 02:14:53.058532 9019 leader_election.cc:341] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:53.059382 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
W20240503 02:14:53.063488 9017 leader_election.cc:341] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:53.063691 9017 leader_election.cc:302] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d; no voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 846e89dd0b154d2b9bb7d4f315c9fc12, cb90f34dc37b4b16b2a5b5a4ee012865
I20240503 02:14:53.065176 10382 raft_consensus.cc:2726] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:53.066684 10453 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:53.066860 10453 ts_tablet_manager.cc:1367] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.007s
I20240503 02:14:53.067641 10453 raft_consensus.cc:348] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.067970 10453 raft_consensus.cc:374] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:53.068053 10453 raft_consensus.cc:729] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:53.068321 10453 consensus_queue.cc:260] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.068984 10453 ts_tablet_manager.cc:1398] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:53.069322 10453 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:53.071134 10453 tablet_bootstrap.cc:654] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:53.072304 10458 ts_tablet_manager.cc:1398] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20240503 02:14:53.072664 10458 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:53.074110 10470 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:53.074268 10458 tablet_bootstrap.cc:654] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:53.074409 10470 ts_tablet_manager.cc:1367] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.006s
I20240503 02:14:53.075269 10470 raft_consensus.cc:348] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.075717 10470 raft_consensus.cc:374] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:53.075954 10470 raft_consensus.cc:729] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:53.076388 10458 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:53.076601 10458 ts_tablet_manager.cc:1367] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:14:53.076624 10470 consensus_queue.cc:260] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.077107 10455 raft_consensus.cc:482] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:53.077250 10455 raft_consensus.cc:504] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.077499 10458 raft_consensus.cc:348] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:53.077884 10458 raft_consensus.cc:374] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:53.077963 10458 raft_consensus.cc:729] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:53.078510 10455 leader_election.cc:288] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:53.078316 10458 consensus_queue.cc:260] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:53.079509 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:53.080111 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:53.080260 10458 ts_tablet_manager.cc:1398] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:14:53.080235 9082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:53.080420 9082 raft_consensus.cc:2454] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
I20240503 02:14:53.080536 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:53.080624 10458 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:53.080744 8558 raft_consensus.cc:2454] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
I20240503 02:14:53.080937 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:53.081390 8950 raft_consensus.cc:2454] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
I20240503 02:14:53.081763 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
W20240503 02:14:53.081923 9148 leader_election.cc:341] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:53.082091 9148 leader_election.cc:302] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d, cb90f34dc37b4b16b2a5b5a4ee012865; no voters: 846e89dd0b154d2b9bb7d4f315c9fc12
I20240503 02:14:53.079751 8819 raft_consensus.cc:2454] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
W20240503 02:14:53.082621 9148 leader_election.cc:341] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:53.082691 9762 raft_consensus.cc:2781] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:53.082782 9762 raft_consensus.cc:482] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:53.082783 10458 tablet_bootstrap.cc:654] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:53.082865 9762 raft_consensus.cc:3037] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.085602 9762 raft_consensus.cc:504] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.086308 10459 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:53.086486 10459 ts_tablet_manager.cc:1367] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.010s
I20240503 02:14:53.087239 10459 raft_consensus.cc:348] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:53.087500 10459 raft_consensus.cc:374] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:53.087575 10459 raft_consensus.cc:729] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:53.087906 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:53.088099 8950 raft_consensus.cc:3037] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.088138 10459 consensus_queue.cc:260] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:53.088748 10459 ts_tablet_manager.cc:1398] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:53.089607 10459 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:53.090703 8950 raft_consensus.cc:2454] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:53.091194 10459 tablet_bootstrap.cc:654] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:53.093408 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:53.093595 8819 raft_consensus.cc:3037] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.093878 10453 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:53.094014 10453 ts_tablet_manager.cc:1367] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.015s
I20240503 02:14:53.094861 10453 raft_consensus.cc:348] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:53.095105 10453 raft_consensus.cc:374] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:53.095187 10453 raft_consensus.cc:729] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:53.095551 10453 consensus_queue.cc:260] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:53.096225 10453 ts_tablet_manager.cc:1398] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:53.096968 9762 leader_election.cc:288] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 election: Requested vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:53.097234 8819 raft_consensus.cc:2454] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:53.097272 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:53.097455 9083 raft_consensus.cc:3037] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.098289 10470 ts_tablet_manager.cc:1398] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240503 02:14:53.098678 10470 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:53.100857 10459 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:53.101361 10459 ts_tablet_manager.cc:1367] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240503 02:14:53.101590 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:53.101944 9083 raft_consensus.cc:2454] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:53.102396 9148 leader_election.cc:302] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:53.102384 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:53.102408 10459 raft_consensus.cc:348] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.102952 10459 raft_consensus.cc:374] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:53.103103 10459 raft_consensus.cc:729] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
W20240503 02:14:53.103247 9148 leader_election.cc:341] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:53.103407 10459 consensus_queue.cc:260] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.104086 10459 ts_tablet_manager.cc:1398] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:14:53.105998 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:53.106209 8558 raft_consensus.cc:3037] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.107445 10470 tablet_bootstrap.cc:654] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:53.108999 8558 raft_consensus.cc:2454] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:53.109129 8689 raft_consensus.cc:3037] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.110457 10470 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:53.110607 10470 ts_tablet_manager.cc:1367] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 02:14:53.111320 10470 raft_consensus.cc:348] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:53.111590 10470 raft_consensus.cc:374] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:53.111691 10470 raft_consensus.cc:729] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:53.111743 8689 raft_consensus.cc:2454] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:53.111958 10470 consensus_queue.cc:260] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:53.112394 10455 raft_consensus.cc:2781] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:53.112634 10470 ts_tablet_manager.cc:1398] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:14:53.113209 10458 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:53.113368 10458 ts_tablet_manager.cc:1367] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.013s
I20240503 02:14:53.113967 10458 raft_consensus.cc:348] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.114166 10458 raft_consensus.cc:374] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:53.114251 10458 raft_consensus.cc:729] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:53.114481 10458 consensus_queue.cc:260] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.115114 10458 ts_tablet_manager.cc:1398] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:14:53.115527 10455 raft_consensus.cc:686] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Becoming Leader. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Running, Role: LEADER
I20240503 02:14:53.115880 10455 consensus_queue.cc:237] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.121213 8278 catalog_manager.cc:5484] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199). New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:53.168921 10484 raft_consensus.cc:482] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:53.169080 10484 raft_consensus.cc:504] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:53.170154 10484 leader_election.cc:288] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747)
I20240503 02:14:53.170318 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:53.170312 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:53.170485 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:53.170555 8689 raft_consensus.cc:2454] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
I20240503 02:14:53.170557 8425 raft_consensus.cc:2454] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
I20240503 02:14:53.170715 8950 raft_consensus.cc:2454] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
I20240503 02:14:53.170683 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:53.170904 8558 raft_consensus.cc:2454] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
I20240503 02:14:53.171198 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:53.171437 9083 raft_consensus.cc:2454] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
I20240503 02:14:53.171380 9149 leader_election.cc:302] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 5 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 846e89dd0b154d2b9bb7d4f315c9fc12, 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:53.171552 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:53.171813 10484 raft_consensus.cc:2781] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:53.171823 8819 raft_consensus.cc:2454] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
I20240503 02:14:53.171928 10484 raft_consensus.cc:482] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:53.172022 10484 raft_consensus.cc:3037] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.173882 10484 raft_consensus.cc:504] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:53.174764 10484 leader_election.cc:288] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 election: Requested vote from peers 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747)
I20240503 02:14:53.175067 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:53.175149 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:53.175277 8425 raft_consensus.cc:3037] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.175257 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:53.175235 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:53.175341 9083 raft_consensus.cc:3037] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.175416 8558 raft_consensus.cc:3037] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.175416 8950 raft_consensus.cc:3037] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.178098 9083 raft_consensus.cc:2454] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:53.178181 8558 raft_consensus.cc:2454] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:53.178174 8950 raft_consensus.cc:2454] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:53.178241 8425 raft_consensus.cc:2454] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:53.178417 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:53.178718 8689 raft_consensus.cc:3037] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.179579 9148 leader_election.cc:302] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:53.179623 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fc93702514499eb32902238c46356d" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:53.179975 10484 raft_consensus.cc:2781] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:53.180008 8819 raft_consensus.cc:3037] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.180228 10484 raft_consensus.cc:686] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Becoming Leader. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Running, Role: LEADER
I20240503 02:14:53.180541 10484 consensus_queue.cc:237] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:53.181396 8689 raft_consensus.cc:2454] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:53.182996 8819 raft_consensus.cc:2454] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 1.
I20240503 02:14:53.184455 8278 catalog_manager.cc:5484] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199). New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } }
I20240503 02:14:53.211472 10478 raft_consensus.cc:482] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:53.211653 10478 raft_consensus.cc:504] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.212833 10478 leader_election.cc:288] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:53.213022 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:53.213104 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:53.213289 9083 raft_consensus.cc:2454] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c668b2e2d5c4edba46cbc5c36065c0c in term 0.
I20240503 02:14:53.213325 8819 raft_consensus.cc:2454] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c668b2e2d5c4edba46cbc5c36065c0c in term 0.
I20240503 02:14:53.213416 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:53.213487 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:53.213661 8558 raft_consensus.cc:2454] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c668b2e2d5c4edba46cbc5c36065c0c in term 0.
I20240503 02:14:53.213718 9213 raft_consensus.cc:2454] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c668b2e2d5c4edba46cbc5c36065c0c in term 0.
I20240503 02:14:53.213801 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:53.214027 8950 raft_consensus.cc:2454] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c668b2e2d5c4edba46cbc5c36065c0c in term 0.
I20240503 02:14:53.214123 8360 leader_election.cc:302] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, c399efcbba924863ac89d156dc07ac7d, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:53.214298 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:53.214493 10478 raft_consensus.cc:2781] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:53.214519 8689 raft_consensus.cc:2454] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c668b2e2d5c4edba46cbc5c36065c0c in term 0.
I20240503 02:14:53.214620 10478 raft_consensus.cc:482] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:53.214725 10478 raft_consensus.cc:3037] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.216903 10478 raft_consensus.cc:504] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.218096 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:53.218163 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:53.218286 9083 raft_consensus.cc:3037] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.218354 9213 raft_consensus.cc:3037] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.218403 10478 leader_election.cc:288] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [CANDIDATE]: Term 1 election: Requested vote from peers 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:53.218741 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:53.218955 8819 raft_consensus.cc:3037] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.218079 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:53.220077 8950 raft_consensus.cc:3037] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.220191 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:53.220400 8558 raft_consensus.cc:3037] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.221202 9083 raft_consensus.cc:2454] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c668b2e2d5c4edba46cbc5c36065c0c in term 1.
I20240503 02:14:53.222178 8819 raft_consensus.cc:2454] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c668b2e2d5c4edba46cbc5c36065c0c in term 1.
I20240503 02:14:53.222324 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea7444883c642419cd3e509f2b8d090" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:53.222518 8689 raft_consensus.cc:3037] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.223289 8558 raft_consensus.cc:2454] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c668b2e2d5c4edba46cbc5c36065c0c in term 1.
I20240503 02:14:53.223726 8950 raft_consensus.cc:2454] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c668b2e2d5c4edba46cbc5c36065c0c in term 1.
I20240503 02:14:53.223940 8358 leader_election.cc:302] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, c399efcbba924863ac89d156dc07ac7d; no voters:
I20240503 02:14:53.224355 10478 raft_consensus.cc:2781] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:53.224463 9213 raft_consensus.cc:2454] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c668b2e2d5c4edba46cbc5c36065c0c in term 1.
I20240503 02:14:53.225152 10478 raft_consensus.cc:686] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 LEADER]: Becoming Leader. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Running, Role: LEADER
I20240503 02:14:53.225387 8689 raft_consensus.cc:2454] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c668b2e2d5c4edba46cbc5c36065c0c in term 1.
I20240503 02:14:53.225811 10478 consensus_queue.cc:237] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.230403 8278 catalog_manager.cc:5484] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193). New cstate: current_term: 1 leader_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:53.361315 10478 raft_consensus.cc:482] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:53.361573 10478 raft_consensus.cc:504] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:14:53.363165 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:53.363214 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:53.363271 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:53.363497 8950 raft_consensus.cc:2379] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c668b2e2d5c4edba46cbc5c36065c0c in current term 1: Already voted for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in this term.
I20240503 02:14:53.363519 8819 raft_consensus.cc:2379] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c668b2e2d5c4edba46cbc5c36065c0c in current term 1: Already voted for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in this term.
I20240503 02:14:53.363760 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:53.364082 9083 raft_consensus.cc:2379] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c668b2e2d5c4edba46cbc5c36065c0c in current term 1: Already voted for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in this term.
I20240503 02:14:53.364475 10478 leader_election.cc:288] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:14:53.364545 8358 leader_election.cc:302] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 4c668b2e2d5c4edba46cbc5c36065c0c; no voters: 79e41f25066e4e5abbd71e48d545c6bf, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d, cb90f34dc37b4b16b2a5b5a4ee012865
I20240503 02:14:53.364895 8558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:53.365003 10478 raft_consensus.cc:3037] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:53.365219 8558 raft_consensus.cc:2379] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c668b2e2d5c4edba46cbc5c36065c0c in current term 1: Already voted for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in this term.
I20240503 02:14:53.365180 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac90f5b4e0fc43e99b3f76fe92610153" candidate_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:53.365468 8689 raft_consensus.cc:2379] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c668b2e2d5c4edba46cbc5c36065c0c in current term 1: Already voted for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in this term.
I20240503 02:14:53.367345 10478 raft_consensus.cc:2726] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:53.497363 10484 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:53.537765 10484 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:53.575191 10484 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:53.582593 10484 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:53.583470 10484 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:53.590504 10484 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:53.601089 10455 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:53.601466 10455 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240503 02:14:53.654413 10489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:53.654862 10489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:14:53.659114 10494 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:53.663290 10484 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:53.665671 10489 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:14:53.666384 10484 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:53.666760 10494 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:53.678048 10478 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:53.726564 10478 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:53.737040 10478 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:53.743757 10478 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:53.764652 10526 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:53.778252 10529 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
INSERT report
rows total: 2000
time total: 412.347 ms
time per row: 0.206174 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:14:55.314378 8275 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42220:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:14:55.315130 8275 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_6 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 02:14:55.323994 8403 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1f333099d942ae897224a1ee1538a6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.324506 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1f333099d942ae897224a1ee1538a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.324442 8670 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1f333099d942ae897224a1ee1538a6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.324887 8670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1f333099d942ae897224a1ee1538a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.324913 8801 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1f333099d942ae897224a1ee1538a6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.325320 8801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1f333099d942ae897224a1ee1538a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.325452 9193 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1f333099d942ae897224a1ee1538a6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.325822 9193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1f333099d942ae897224a1ee1538a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.328182 8538 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1f333099d942ae897224a1ee1538a6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.328621 8538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1f333099d942ae897224a1ee1538a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.328603 8932 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1f333099d942ae897224a1ee1538a6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.329003 8932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1f333099d942ae897224a1ee1538a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.332299 9062 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1f333099d942ae897224a1ee1538a6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.332821 9062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1f333099d942ae897224a1ee1538a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.339306 10598 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:55.341935 10598 tablet_bootstrap.cc:654] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.345501 10597 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:55.349103 10597 tablet_bootstrap.cc:654] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.350750 8669 tablet_service.cc:1467] Processing CreateTablet for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.351174 8669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.352069 8404 tablet_service.cc:1467] Processing CreateTablet for tablet 1a30758cbd1d4845bb9eeb02e1a2942a (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.352427 8404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a30758cbd1d4845bb9eeb02e1a2942a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.352383 8668 tablet_service.cc:1467] Processing CreateTablet for tablet 1a30758cbd1d4845bb9eeb02e1a2942a (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.352661 8538 tablet_service.cc:1467] Processing CreateTablet for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.352774 8668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a30758cbd1d4845bb9eeb02e1a2942a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.353035 8538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.355664 9063 tablet_service.cc:1467] Processing CreateTablet for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.356058 9063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.356439 9192 tablet_service.cc:1467] Processing CreateTablet for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.356804 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.358624 8403 tablet_service.cc:1467] Processing CreateTablet for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.358989 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.360194 8537 tablet_service.cc:1467] Processing CreateTablet for tablet 1a30758cbd1d4845bb9eeb02e1a2942a (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.360594 8537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a30758cbd1d4845bb9eeb02e1a2942a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.364230 8800 tablet_service.cc:1467] Processing CreateTablet for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.364612 8800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.368829 8931 tablet_service.cc:1467] Processing CreateTablet for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.369215 8931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.378027 8799 tablet_service.cc:1467] Processing CreateTablet for tablet 1a30758cbd1d4845bb9eeb02e1a2942a (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.378454 8799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a30758cbd1d4845bb9eeb02e1a2942a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.379261 8930 tablet_service.cc:1467] Processing CreateTablet for tablet 1a30758cbd1d4845bb9eeb02e1a2942a (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.379913 8930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a30758cbd1d4845bb9eeb02e1a2942a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.379832 9193 tablet_service.cc:1467] Processing CreateTablet for tablet 1a30758cbd1d4845bb9eeb02e1a2942a (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.380259 9193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a30758cbd1d4845bb9eeb02e1a2942a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.382696 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 1a30758cbd1d4845bb9eeb02e1a2942a (DEFAULT_TABLE table=rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:55.383184 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a30758cbd1d4845bb9eeb02e1a2942a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:55.399004 10602 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:55.400326 10604 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:55.401008 10602 tablet_bootstrap.cc:654] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.402431 10603 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:55.402477 10597 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:55.403350 10597 ts_tablet_manager.cc:1367] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.058s user 0.000s sys 0.008s
I20240503 02:14:55.404443 10603 tablet_bootstrap.cc:654] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.404446 10597 raft_consensus.cc:348] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.405959 10598 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:55.406162 10598 ts_tablet_manager.cc:1367] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.000s
I20240503 02:14:55.407194 10598 raft_consensus.cc:348] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.407652 10598 raft_consensus.cc:374] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.407821 10598 raft_consensus.cc:729] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.408193 10598 consensus_queue.cc:260] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.409847 10604 tablet_bootstrap.cc:654] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.410585 10598 ts_tablet_manager.cc:1398] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:14:55.410848 10599 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:55.410967 10598 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:55.412515 10599 tablet_bootstrap.cc:654] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.412745 10602 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:55.412982 10602 ts_tablet_manager.cc:1367] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20240503 02:14:55.412961 10597 raft_consensus.cc:374] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.413074 10597 raft_consensus.cc:729] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.413427 10597 consensus_queue.cc:260] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.414223 10598 tablet_bootstrap.cc:654] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.414250 10603 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:55.414381 10603 ts_tablet_manager.cc:1367] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 02:14:55.413851 10602 raft_consensus.cc:348] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.414975 10602 raft_consensus.cc:374] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.415122 10602 raft_consensus.cc:729] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.415123 10603 raft_consensus.cc:348] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.415470 10603 raft_consensus.cc:374] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.415500 10597 ts_tablet_manager.cc:1398] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:14:55.415594 10603 raft_consensus.cc:729] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.415401 10602 consensus_queue.cc:260] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.415906 10597 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:55.415926 10603 consensus_queue.cc:260] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.416149 10602 ts_tablet_manager.cc:1398] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:14:55.416452 10602 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:55.416646 10603 ts_tablet_manager.cc:1398] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:55.416908 10603 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:55.418602 10603 tablet_bootstrap.cc:654] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.419072 10597 tablet_bootstrap.cc:654] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.419811 10596 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:55.422883 10596 tablet_bootstrap.cc:654] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.423501 10602 tablet_bootstrap.cc:654] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.424120 10604 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:55.424290 10604 ts_tablet_manager.cc:1367] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.005s
I20240503 02:14:55.424472 10598 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:55.424758 10598 ts_tablet_manager.cc:1367] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20240503 02:14:55.425738 10604 raft_consensus.cc:348] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.426061 10604 raft_consensus.cc:374] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.426182 10604 raft_consensus.cc:729] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.425879 10598 raft_consensus.cc:348] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.426546 10598 raft_consensus.cc:374] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.426412 10604 consensus_queue.cc:260] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.426685 10598 raft_consensus.cc:729] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.427091 10604 ts_tablet_manager.cc:1398] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:14:55.429083 10599 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:55.429265 10599 ts_tablet_manager.cc:1367] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20240503 02:14:55.429706 10598 consensus_queue.cc:260] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.430002 10599 raft_consensus.cc:348] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.430357 10599 raft_consensus.cc:374] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.430534 10599 raft_consensus.cc:729] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.430831 10598 ts_tablet_manager.cc:1398] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:14:55.430852 10599 consensus_queue.cc:260] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.431432 10598 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:14:55.432543 10599 ts_tablet_manager.cc:1398] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 02:14:55.432878 10599 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:55.435245 10599 tablet_bootstrap.cc:654] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.436450 10603 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:55.436676 10603 ts_tablet_manager.cc:1367] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240503 02:14:55.437925 10201 raft_consensus.cc:482] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:55.438171 10201 raft_consensus.cc:504] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.439486 10598 tablet_bootstrap.cc:654] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.439877 10201 leader_election.cc:288] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:55.439972 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:55.437633 10603 raft_consensus.cc:348] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.440528 10603 raft_consensus.cc:374] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.440618 10603 raft_consensus.cc:729] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.440706 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:55.440871 10603 consensus_queue.cc:260] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
W20240503 02:14:55.441268 8493 leader_election.cc:341] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:55.441349 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:55.441489 10603 ts_tablet_manager.cc:1398] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20240503 02:14:55.441641 8492 leader_election.cc:341] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:55.441661 8424 raft_consensus.cc:2454] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:55.442111 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:55.442610 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:55.442854 8819 raft_consensus.cc:2454] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
I20240503 02:14:55.442921 10596 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:55.443167 10596 ts_tablet_manager.cc:1367] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.005s
I20240503 02:14:55.443432 10603 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d: Bootstrap starting.
I20240503 02:14:55.443460 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:55.443688 9083 raft_consensus.cc:2454] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 0.
W20240503 02:14:55.445096 8491 leader_election.cc:341] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:55.445325 8491 leader_election.cc:302] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 79e41f25066e4e5abbd71e48d545c6bf, c399efcbba924863ac89d156dc07ac7d; no voters: 846e89dd0b154d2b9bb7d4f315c9fc12, 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865
I20240503 02:14:55.445449 10603 tablet_bootstrap.cc:654] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.444170 10596 raft_consensus.cc:348] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.446027 10596 raft_consensus.cc:374] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.446135 10445 raft_consensus.cc:2781] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:55.446187 10596 raft_consensus.cc:729] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.446246 10445 raft_consensus.cc:482] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:55.446346 10445 raft_consensus.cc:3037] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.448280 10596 consensus_queue.cc:260] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.448897 10598 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431: No bootstrap required, opened a new log
I20240503 02:14:55.449061 10598 ts_tablet_manager.cc:1367] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240503 02:14:55.449216 10596 ts_tablet_manager.cc:1398] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:14:55.449218 10445 raft_consensus.cc:504] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.449800 10596 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:55.450771 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
W20240503 02:14:55.451710 8492 leader_election.cc:341] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:55.452308 10445 leader_election.cc:288] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 election: Requested vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:55.452672 10598 raft_consensus.cc:348] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.453054 10598 raft_consensus.cc:374] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.453173 10598 raft_consensus.cc:729] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.454793 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:55.454919 10596 tablet_bootstrap.cc:654] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.455004 8689 raft_consensus.cc:3037] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.454850 10598 consensus_queue.cc:260] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.456696 10598 ts_tablet_manager.cc:1398] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 02:14:55.457765 8689 raft_consensus.cc:2454] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:55.457988 10599 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:55.458184 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:55.458277 10599 ts_tablet_manager.cc:1367] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.002s
I20240503 02:14:55.458408 8819 raft_consensus.cc:3037] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.459362 10599 raft_consensus.cc:348] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.460011 10599 raft_consensus.cc:374] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.460238 10599 raft_consensus.cc:729] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.460922 10597 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:55.461277 10599 consensus_queue.cc:260] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.462409 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:55.462909 10599 ts_tablet_manager.cc:1398] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:14:55.462561 10597 ts_tablet_manager.cc:1367] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.047s user 0.018s sys 0.007s
W20240503 02:14:55.463131 8493 leader_election.cc:341] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:14:55.463243 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:55.463466 8424 raft_consensus.cc:3037] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.464171 10599 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:14:55.464413 10603 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d: No bootstrap required, opened a new log
I20240503 02:14:55.464576 10603 ts_tablet_manager.cc:1367] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.000s
I20240503 02:14:55.465085 10597 raft_consensus.cc:348] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.465369 10597 raft_consensus.cc:374] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.465245 10603 raft_consensus.cc:348] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.465449 10597 raft_consensus.cc:729] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.465477 10603 raft_consensus.cc:374] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.465576 10603 raft_consensus.cc:729] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.465826 10603 consensus_queue.cc:260] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.466207 8424 raft_consensus.cc:2454] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:55.467228 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "2a1893ab239f4163a06bdbf8ce579431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:55.467417 9083 raft_consensus.cc:3037] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.467360 10597 consensus_queue.cc:260] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.467998 10597 ts_tablet_manager.cc:1398] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:14:55.468318 10597 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:14:55.469928 10597 tablet_bootstrap.cc:654] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.470183 9083 raft_consensus.cc:2454] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:55.470948 10599 tablet_bootstrap.cc:654] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.472628 10596 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:55.472857 10596 ts_tablet_manager.cc:1367] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.001s
I20240503 02:14:55.473654 10596 raft_consensus.cc:348] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.474045 10596 raft_consensus.cc:374] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.474174 10596 raft_consensus.cc:729] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.474371 10603 ts_tablet_manager.cc:1398] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:55.474556 10596 consensus_queue.cc:260] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.475383 10596 ts_tablet_manager.cc:1398] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:55.476361 10596 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
I20240503 02:14:55.476975 8491 leader_election.cc:302] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 846e89dd0b154d2b9bb7d4f315c9fc12, c399efcbba924863ac89d156dc07ac7d; no voters: 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865
I20240503 02:14:55.477267 10618 raft_consensus.cc:2781] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:55.477402 10618 raft_consensus.cc:686] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Becoming Leader. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Running, Role: LEADER
I20240503 02:14:55.477717 10618 consensus_queue.cc:237] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.461277 8819 raft_consensus.cc:2454] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1893ab239f4163a06bdbf8ce579431 in term 1.
I20240503 02:14:55.483194 10604 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:55.485484 10596 tablet_bootstrap.cc:654] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.486817 10599 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf: No bootstrap required, opened a new log
I20240503 02:14:55.486946 10597 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c: No bootstrap required, opened a new log
I20240503 02:14:55.486968 10599 ts_tablet_manager.cc:1367] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.009s
I20240503 02:14:55.487154 10597 ts_tablet_manager.cc:1367] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.002s
I20240503 02:14:55.487735 10599 raft_consensus.cc:348] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.487977 10599 raft_consensus.cc:374] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.488076 10599 raft_consensus.cc:729] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.488346 10599 consensus_queue.cc:260] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.489070 10599 ts_tablet_manager.cc:1398] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:14:55.488781 8282 catalog_manager.cc:5484] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194). New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:55.490674 10604 tablet_bootstrap.cc:654] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.491968 10597 raft_consensus.cc:348] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.492328 10597 raft_consensus.cc:374] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.492436 10597 raft_consensus.cc:729] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c668b2e2d5c4edba46cbc5c36065c0c, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.493930 10604 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:55.494102 10604 ts_tablet_manager.cc:1367] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 02:14:55.494863 10604 raft_consensus.cc:348] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.495169 10604 raft_consensus.cc:374] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.495240 10604 raft_consensus.cc:729] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.497296 10604 consensus_queue.cc:260] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.498648 10604 ts_tablet_manager.cc:1398] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:14:55.499753 10602 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:55.499966 10602 ts_tablet_manager.cc:1367] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.084s user 0.004s sys 0.006s
I20240503 02:14:55.500828 10602 raft_consensus.cc:348] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.501116 10602 raft_consensus.cc:374] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.501202 10602 raft_consensus.cc:729] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.501555 10602 consensus_queue.cc:260] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.502200 10602 ts_tablet_manager.cc:1398] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:14:55.502487 10602 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865: Bootstrap starting.
I20240503 02:14:55.492720 10597 consensus_queue.cc:260] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.504112 10602 tablet_bootstrap.cc:654] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.505671 10597 ts_tablet_manager.cc:1398] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240503 02:14:55.507287 10604 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:14:55.508821 10604 tablet_bootstrap.cc:654] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:55.511180 10596 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12: No bootstrap required, opened a new log
I20240503 02:14:55.511410 10596 ts_tablet_manager.cc:1367] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.012s
I20240503 02:14:55.512359 10596 raft_consensus.cc:348] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.512712 10596 raft_consensus.cc:374] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.512742 10602 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865: No bootstrap required, opened a new log
I20240503 02:14:55.512823 10596 raft_consensus.cc:729] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.512950 10602 ts_tablet_manager.cc:1367] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 02:14:55.513875 10596 consensus_queue.cc:260] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.514712 10596 ts_tablet_manager.cc:1398] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:14:55.518167 10602 raft_consensus.cc:348] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.518951 10602 raft_consensus.cc:374] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.519093 10602 raft_consensus.cc:729] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb90f34dc37b4b16b2a5b5a4ee012865, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.519523 10604 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: No bootstrap required, opened a new log
I20240503 02:14:55.519381 10602 consensus_queue.cc:260] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.519830 10604 ts_tablet_manager.cc:1367] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20240503 02:14:55.520032 10602 ts_tablet_manager.cc:1398] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 02:14:55.520576 10604 raft_consensus.cc:348] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.520864 10604 raft_consensus.cc:374] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:55.520965 10604 raft_consensus.cc:729] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: FOLLOWER
I20240503 02:14:55.521188 10604 consensus_queue.cc:260] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.521802 10604 ts_tablet_manager.cc:1398] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:55.531747 10382 raft_consensus.cc:482] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:55.531963 10382 raft_consensus.cc:504] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.533113 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a30758cbd1d4845bb9eeb02e1a2942a" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:55.533236 10382 leader_election.cc:288] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:55.533372 8424 raft_consensus.cc:2454] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:55.533339 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a30758cbd1d4845bb9eeb02e1a2942a" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:55.533567 8690 raft_consensus.cc:2454] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:55.533905 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a30758cbd1d4845bb9eeb02e1a2942a" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:55.533998 8556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a30758cbd1d4845bb9eeb02e1a2942a" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:55.534102 8949 raft_consensus.cc:2454] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:55.534237 8556 raft_consensus.cc:2454] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:55.534440 9018 leader_election.cc:302] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 4c668b2e2d5c4edba46cbc5c36065c0c, 846e89dd0b154d2b9bb7d4f315c9fc12, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d; no voters:
I20240503 02:14:55.534709 10382 raft_consensus.cc:2781] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:55.534822 10382 raft_consensus.cc:482] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:55.534905 10382 raft_consensus.cc:3037] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.535045 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a30758cbd1d4845bb9eeb02e1a2942a" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:55.535080 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a30758cbd1d4845bb9eeb02e1a2942a" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:55.535297 9213 raft_consensus.cc:2454] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:55.535338 8819 raft_consensus.cc:2454] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 0.
I20240503 02:14:55.536948 10382 raft_consensus.cc:504] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.537854 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a30758cbd1d4845bb9eeb02e1a2942a" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:55.537948 8556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a30758cbd1d4845bb9eeb02e1a2942a" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:55.538048 8424 raft_consensus.cc:3037] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.538064 10382 leader_election.cc:288] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 election: Requested vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:55.538138 8556 raft_consensus.cc:3037] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.538233 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a30758cbd1d4845bb9eeb02e1a2942a" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:55.538425 8690 raft_consensus.cc:3037] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.540306 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a30758cbd1d4845bb9eeb02e1a2942a" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:55.540455 9213 raft_consensus.cc:3037] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.540932 8424 raft_consensus.cc:2454] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:55.541175 8690 raft_consensus.cc:2454] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:55.541505 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a30758cbd1d4845bb9eeb02e1a2942a" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3"
I20240503 02:14:55.541713 8949 raft_consensus.cc:3037] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.542376 9213 raft_consensus.cc:2454] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:55.543015 8556 raft_consensus.cc:2454] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:55.543972 8949 raft_consensus.cc:2454] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:55.544050 9017 leader_election.cc:302] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 846e89dd0b154d2b9bb7d4f315c9fc12, c399efcbba924863ac89d156dc07ac7d, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:55.544317 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a30758cbd1d4845bb9eeb02e1a2942a" candidate_uuid: "c399efcbba924863ac89d156dc07ac7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:55.544561 8819 raft_consensus.cc:3037] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.544555 10382 raft_consensus.cc:2781] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:55.544698 10382 raft_consensus.cc:686] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Becoming Leader. State: Replica: c399efcbba924863ac89d156dc07ac7d, State: Running, Role: LEADER
I20240503 02:14:55.545029 10382 consensus_queue.cc:237] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.548491 8819 raft_consensus.cc:2454] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c399efcbba924863ac89d156dc07ac7d in term 1.
I20240503 02:14:55.549624 8275 catalog_manager.cc:5484] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c399efcbba924863ac89d156dc07ac7d (127.23.104.198). New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:55.586033 10590 raft_consensus.cc:482] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:55.586179 10590 raft_consensus.cc:504] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.587141 10590 leader_election.cc:288] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:55.587514 8820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c28f0e30cfe46fda64dd2b0fd7847c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:55.587725 8820 raft_consensus.cc:2454] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:55.587731 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c28f0e30cfe46fda64dd2b0fd7847c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" is_pre_election: true
I20240503 02:14:55.587824 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c28f0e30cfe46fda64dd2b0fd7847c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:55.587838 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c28f0e30cfe46fda64dd2b0fd7847c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:55.587993 9213 raft_consensus.cc:2454] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:55.588030 9083 raft_consensus.cc:2454] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:55.588042 8424 raft_consensus.cc:2454] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:55.588380 8557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c28f0e30cfe46fda64dd2b0fd7847c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:55.588682 8557 raft_consensus.cc:2454] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:55.589080 8888 leader_election.cc:302] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 4c668b2e2d5c4edba46cbc5c36065c0c, 99be4ff1114d4aaabcc537604fce54b3, c399efcbba924863ac89d156dc07ac7d, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:55.589084 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c28f0e30cfe46fda64dd2b0fd7847c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:55.589311 8690 raft_consensus.cc:2454] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 0.
I20240503 02:14:55.589380 10590 raft_consensus.cc:2781] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:55.589495 10590 raft_consensus.cc:482] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:55.589581 10590 raft_consensus.cc:3037] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.591568 10590 raft_consensus.cc:504] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.592983 8557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c28f0e30cfe46fda64dd2b0fd7847c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431"
I20240503 02:14:55.593015 9213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c28f0e30cfe46fda64dd2b0fd7847c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865"
I20240503 02:14:55.593024 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c28f0e30cfe46fda64dd2b0fd7847c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c"
I20240503 02:14:55.593189 8557 raft_consensus.cc:3037] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.593218 9213 raft_consensus.cc:3037] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.593232 8424 raft_consensus.cc:3037] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.593722 8820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c28f0e30cfe46fda64dd2b0fd7847c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf"
I20240503 02:14:55.593902 8820 raft_consensus.cc:3037] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.595118 10590 leader_election.cc:288] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 election: Requested vote from peers 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237), cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:14:55.596000 8557 raft_consensus.cc:2454] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:55.595984 8424 raft_consensus.cc:2454] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:55.596218 9213 raft_consensus.cc:2454] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:55.596593 8820 raft_consensus.cc:2454] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:55.597141 8888 leader_election.cc:302] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 2a1893ab239f4163a06bdbf8ce579431, 4c668b2e2d5c4edba46cbc5c36065c0c, 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865; no voters:
I20240503 02:14:55.597251 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c28f0e30cfe46fda64dd2b0fd7847c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12"
I20240503 02:14:55.597445 10511 raft_consensus.cc:2781] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:55.597458 8690 raft_consensus.cc:3037] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.597554 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c28f0e30cfe46fda64dd2b0fd7847c6" candidate_uuid: "99be4ff1114d4aaabcc537604fce54b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d"
I20240503 02:14:55.597755 9083 raft_consensus.cc:3037] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.600551 9083 raft_consensus.cc:2454] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:55.601517 8690 raft_consensus.cc:2454] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99be4ff1114d4aaabcc537604fce54b3 in term 1.
I20240503 02:14:55.602244 10511 raft_consensus.cc:686] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Becoming Leader. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Running, Role: LEADER
I20240503 02:14:55.602587 10511 consensus_queue.cc:237] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.606065 8275 catalog_manager.cc:5484] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197). New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } }
I20240503 02:14:55.701776 10510 raft_consensus.cc:482] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:55.701977 10510 raft_consensus.cc:504] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:14:55.703188 10510 leader_election.cc:288] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635), 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923), 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495), 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403), 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747), c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:14:55.703886 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99be4ff1114d4aaabcc537604fce54b3" is_pre_election: true
I20240503 02:14:55.704106 8949 raft_consensus.cc:2454] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb90f34dc37b4b16b2a5b5a4ee012865 in term 0.
I20240503 02:14:55.704938 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" is_pre_election: true
I20240503 02:14:55.705077 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c399efcbba924863ac89d156dc07ac7d" is_pre_election: true
I20240503 02:14:55.705231 8690 raft_consensus.cc:2379] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cb90f34dc37b4b16b2a5b5a4ee012865 in current term 1: Already voted for candidate 2a1893ab239f4163a06bdbf8ce579431 in this term.
I20240503 02:14:55.705341 9083 raft_consensus.cc:2379] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cb90f34dc37b4b16b2a5b5a4ee012865 in current term 1: Already voted for candidate 2a1893ab239f4163a06bdbf8ce579431 in this term.
I20240503 02:14:55.706144 8820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e41f25066e4e5abbd71e48d545c6bf" is_pre_election: true
I20240503 02:14:55.706429 8820 raft_consensus.cc:2379] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cb90f34dc37b4b16b2a5b5a4ee012865 in current term 1: Already voted for candidate 2a1893ab239f4163a06bdbf8ce579431 in this term.
I20240503 02:14:55.707231 8557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1893ab239f4163a06bdbf8ce579431" is_pre_election: true
I20240503 02:14:55.708072 9148 leader_election.cc:302] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 99be4ff1114d4aaabcc537604fce54b3, cb90f34dc37b4b16b2a5b5a4ee012865; no voters: 2a1893ab239f4163a06bdbf8ce579431, 79e41f25066e4e5abbd71e48d545c6bf, 846e89dd0b154d2b9bb7d4f315c9fc12, c399efcbba924863ac89d156dc07ac7d
I20240503 02:14:55.709735 10510 raft_consensus.cc:3037] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:55.711992 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1f333099d942ae897224a1ee1538a6" candidate_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" is_pre_election: true
I20240503 02:14:55.712260 8424 raft_consensus.cc:2379] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cb90f34dc37b4b16b2a5b5a4ee012865 in current term 1: Already voted for candidate 2a1893ab239f4163a06bdbf8ce579431 in this term.
I20240503 02:14:55.713073 10510 raft_consensus.cc:2726] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:14:55.872994 10445 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:55.903487 10618 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:55.957783 10448 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:55.990537 10511 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:56.011336 10448 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:56.014932 10628 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:56.015529 10628 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:56.017292 8950 raft_consensus.cc:3037] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:56.025791 10618 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:56.038767 10618 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:56.053471 10590 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:56.068830 10445 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:56.083133 10590 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:56.093134 10590 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:56.102170 10445 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:56.140587 10382 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:56.141717 10382 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:56.150002 10382 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:56.187491 10664 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:56.225198 10630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:56.225587 10630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:56.233109 10630 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
INSERT report
rows total: 2000
time total: 396.32 ms
time per row: 0.19816 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:14:57.785917 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.200:45027
--local_ip_for_outbound_sockets=127.23.104.200
--tserver_master_addrs=127.23.104.254:36549
--webserver_port=44021
--webserver_interface=127.23.104.200
--builtin_ntp_servers=127.23.104.212:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:14:58.139359 10734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:58.139652 10734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:58.139842 10734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:58.147521 10734 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:58.147634 10734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:58.147827 10734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.200
I20240503 02:14:58.156563 10734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.200:45027
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.23.104.200
--webserver_port=44021
--tserver_master_addrs=127.23.104.254:36549
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.200
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:58.157878 10734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:58.159544 10734 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:58.171365 10740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:58.171962 10742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:58.172394 10739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:58.172750 10734 server_base.cc:1034] running on GCE node
I20240503 02:14:58.173259 10734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:58.174098 10734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:58.175311 10734 hybrid_clock.cc:648] HybridClock initialized: now 1714702498175262 us; error 49 us; skew 500 ppm
I20240503 02:14:58.175592 10734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:58.177691 10734 webserver.cc:460] Webserver started at http://127.23.104.200:44021/ using document root <none> and password file <none>
I20240503 02:14:58.178481 10734 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:58.178611 10734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:58.184419 10734 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:14:58.187260 10748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:58.188613 10734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:14:58.188781 10734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df"
format_stamp: "Formatted at 2024-05-03 02:14:37 on dist-test-slave-38ht"
I20240503 02:14:58.189604 10734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:58.208926 10734 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:58.209616 10734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:58.209867 10734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:58.210693 10734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:58.212092 10734 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:58.212174 10734 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:58.212299 10734 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:58.212342 10734 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:58.258271 10734 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.200:45027
I20240503 02:14:58.258322 10860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.200:45027 every 8 connection(s)
I20240503 02:14:58.259836 10734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:14:58.265190 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 10734
I20240503 02:14:58.265825 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.201:46877
--local_ip_for_outbound_sockets=127.23.104.201
--tserver_master_addrs=127.23.104.254:36549
--webserver_port=34709
--webserver_interface=127.23.104.201
--builtin_ntp_servers=127.23.104.212:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:58.273617 10861 heartbeater.cc:347] Connected to a master server at 127.23.104.254:36549
I20240503 02:14:58.273900 10861 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:58.274668 10861 heartbeater.cc:511] Master 127.23.104.254:36549 requested a full tablet report, sending...
I20240503 02:14:58.276544 8275 ts_manager.cc:194] Re-registered known tserver with Master: e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200:45027)
I20240503 02:14:58.277496 8275 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.200:33211
W20240503 02:14:58.628549 10865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:58.628847 10865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:58.628980 10865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:58.637349 10865 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:58.637496 10865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:58.637693 10865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.201
I20240503 02:14:58.650061 10865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.201:46877
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.23.104.201
--webserver_port=34709
--tserver_master_addrs=127.23.104.254:36549
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.201
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:58.651427 10865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:58.653116 10865 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:58.668045 10870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:58.668668 10873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:58.669096 10871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:58.669461 10865 server_base.cc:1034] running on GCE node
I20240503 02:14:58.669997 10865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:58.670890 10865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:58.672082 10865 hybrid_clock.cc:648] HybridClock initialized: now 1714702498672035 us; error 41 us; skew 500 ppm
I20240503 02:14:58.672412 10865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:58.677747 10865 webserver.cc:460] Webserver started at http://127.23.104.201:34709/ using document root <none> and password file <none>
I20240503 02:14:58.678498 10865 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:58.678642 10865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:58.684480 10865 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 02:14:58.692456 10879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:58.693958 10865 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20240503 02:14:58.694123 10865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "4792082063bd4d58bc222c9c95b16335"
format_stamp: "Formatted at 2024-05-03 02:14:37 on dist-test-slave-38ht"
I20240503 02:14:58.695008 10865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:58.710304 10865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:58.711133 10865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:58.711441 10865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:58.712527 10865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:58.714202 10865 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:58.714290 10865 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:58.714423 10865 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:58.714493 10865 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:58.749471 10865 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.201:46877
I20240503 02:14:58.749503 10991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.201:46877 every 8 connection(s)
I20240503 02:14:58.750881 10865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240503 02:14:58.758396 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 10865
I20240503 02:14:58.759187 23971 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
/tmp/dist-test-taskTHQSCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.104.202:37759
--local_ip_for_outbound_sockets=127.23.104.202
--tserver_master_addrs=127.23.104.254:36549
--webserver_port=39537
--webserver_interface=127.23.104.202
--builtin_ntp_servers=127.23.104.212:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:14:58.764495 10992 heartbeater.cc:347] Connected to a master server at 127.23.104.254:36549
I20240503 02:14:58.764978 10992 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:58.766053 10992 heartbeater.cc:511] Master 127.23.104.254:36549 requested a full tablet report, sending...
I20240503 02:14:58.767552 8275 ts_manager.cc:194] Re-registered known tserver with Master: 4792082063bd4d58bc222c9c95b16335 (127.23.104.201:46877)
I20240503 02:14:58.768757 8275 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.201:33047
W20240503 02:14:59.121512 10995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:59.121917 10995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:59.122072 10995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:59.130203 10995 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:14:59.130344 10995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:59.130524 10995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.104.202
I20240503 02:14:59.140659 10995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.104.212:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.104.202:37759
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.23.104.202
--webserver_port=39537
--tserver_master_addrs=127.23.104.254:36549
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.104.202
--log_dir=/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:59.141938 10995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:59.143833 10995 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:59.156450 11002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:59.156962 11005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:59.157485 11003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:59.157976 10995 server_base.cc:1034] running on GCE node
I20240503 02:14:59.158576 10995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:59.159504 10995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:59.160765 10995 hybrid_clock.cc:648] HybridClock initialized: now 1714702499160717 us; error 91 us; skew 500 ppm
I20240503 02:14:59.161137 10995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:59.163662 10995 webserver.cc:460] Webserver started at http://127.23.104.202:39537/ using document root <none> and password file <none>
I20240503 02:14:59.164656 10995 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:59.164811 10995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:59.169708 10995 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 02:14:59.172426 11011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:59.173624 10995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:14:59.173775 10995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "dabeea74472746538132a520710248ea"
format_stamp: "Formatted at 2024-05-03 02:14:38 on dist-test-slave-38ht"
I20240503 02:14:59.174510 10995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:59.212847 10995 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:14:59.213779 10995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:59.214026 10995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:59.214917 10995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:59.216334 10995 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:59.216405 10995 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:59.216492 10995 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:59.216533 10995 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:14:59.268247 10995 rpc_server.cc:307] RPC server started. Bound to: 127.23.104.202:37759
I20240503 02:14:59.268293 11123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.104.202:37759 every 8 connection(s)
I20240503 02:14:59.269570 10995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240503 02:14:59.273064 23971 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTHQSCq/build/asan/bin/kudu as pid 10995
I20240503 02:14:59.280143 10861 heartbeater.cc:503] Master 127.23.104.254:36549 was elected leader, sending a full tablet report...
I20240503 02:14:59.282012 11124 heartbeater.cc:347] Connected to a master server at 127.23.104.254:36549
I20240503 02:14:59.282285 11124 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:59.283051 11124 heartbeater.cc:511] Master 127.23.104.254:36549 requested a full tablet report, sending...
I20240503 02:14:59.284987 8275 ts_manager.cc:194] Re-registered known tserver with Master: dabeea74472746538132a520710248ea (127.23.104.202:37759)
I20240503 02:14:59.285660 8275 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.104.202:59541
I20240503 02:14:59.771692 10992 heartbeater.cc:503] Master 127.23.104.254:36549 was elected leader, sending a full tablet report...
I20240503 02:14:59.811555 8537 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:59.818830 10927 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:59.820876 8403 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:59.852007 8801 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:59.899708 8670 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:59.905678 8931 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:59.938160 9193 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:59.945273 9062 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:59.962216 10796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:59.963030 11059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:00.203647 8537 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:00.204691 10927 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:00.227686 8403 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:00.233592 8801 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:15:00.254099 8931 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:15:00.264818 8670 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:00.289497 11124 heartbeater.cc:503] Master 127.23.104.254:36549 was elected leader, sending a full tablet report...
I20240503 02:15:00.314898 9062 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:00.321647 11059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:00.330469 10796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:00.331632 9193 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:15:00.622045 8537 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:00.645045 8403 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:00.648356 10927 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:00.654677 11059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:00.665153 9193 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:15:00.689702 8931 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:15:00.699106 10796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:00.704083 8670 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:00.712494 8801 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:15:00.712659 9062 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:00.863584 9082 consensus_queue.cc:237] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:15:00.867214 9213 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:00.867396 8424 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:00.867396 8949 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:00.867753 11278 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:00.868172 11129 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:00.868518 8819 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:00.868579 8557 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:00.868810 11129 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:00.869097 10997 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:00.869482 8690 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:00.869524 11279 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:00.870102 11279 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:00.874490 11278 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:00.876073 8819 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:00.875959 8557 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:00.879179 8424 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:00.881206 8949 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:00.881645 9213 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:00.885080 8690 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:00.887302 8281 catalog_manager.cc:5484] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:00.908141 9212 consensus_queue.cc:237] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:15:00.911346 8424 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:00.911346 8557 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:00.911381 8690 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:00.912124 11171 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:00.912446 10510 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:00.913702 9082 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:00.913785 8949 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:00.914317 11297 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:00.914594 10510 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:00.915935 11295 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:00.916057 8819 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:00.916875 11295 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:00.943336 11295 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:00.947163 9083 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:00.948465 8425 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:00.952491 8690 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:00.952608 8819 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:00.957600 8556 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
W20240503 02:15:00.958683 9149 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:45403, user_credentials={real_user=slave}} blocked reactor thread for 44341.4us
I20240503 02:15:00.961195 8949 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:00.971041 9082 consensus_queue.cc:237] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:00.975999 9212 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:00.976157 8819 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:00.976806 8425 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:00.976770 11279 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:00.977571 8556 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:00.978019 8949 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:00.978227 11279 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:00.978533 10997 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:00.981976 8274 catalog_manager.cc:5484] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:00.984973 10997 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:00.989101 8267 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet df68cc6032b14cae9e1cac074976baf8 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:00.989681 8950 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:00.992408 8821 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:00.999727 9212 consensus_queue.cc:237] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:01.003010 11279 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.003577 11279 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.004976 9083 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.005198 8819 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.006850 8556 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.007716 8949 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.008344 10510 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.008814 10510 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.009243 10510 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.009464 8689 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.010203 8424 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.013546 8281 catalog_manager.cc:5484] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:01.013566 8556 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:01.015177 11296 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.015627 11296 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.019397 11295 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:01.020181 8689 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
W20240503 02:15:01.024355 9019 consensus_peers.cc:485] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d -> Peer e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200:45027): Couldn't send request to peer e512cb0d8f244ac6ab0e7150a0a8e2df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df68cc6032b14cae9e1cac074976baf8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:00.998080 8425 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:01.028239 9083 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:01.029237 8273 catalog_manager.cc:5484] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:01.032802 8269 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 879167ff15db41a7b4ef21fa10c44b27 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:01.033510 11296 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.033574 9212 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:01.036116 8819 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:01.037729 8556 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:01.026927 8424 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:01.049117 8690 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.050208 10997 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.053627 8690 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
W20240503 02:15:01.060513 9150 consensus_peers.cc:485] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 -> Peer e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200:45027): Couldn't send request to peer e512cb0d8f244ac6ab0e7150a0a8e2df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 879167ff15db41a7b4ef21fa10c44b27. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:01.061551 8949 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:01.073966 8821 consensus_queue.cc:237] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } }
I20240503 02:15:01.077265 8949 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.078334 9083 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.079195 10996 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.079728 10996 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.081305 9212 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.081938 8556 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.082386 8425 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.081895 11316 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.083141 11317 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.083526 11315 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.083827 8690 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.084473 11315 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.089051 11315 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } }
I20240503 02:15:01.089758 8689 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } }
I20240503 02:15:01.093099 8556 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } }
I20240503 02:15:01.093600 8425 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } }
I20240503 02:15:01.100245 8281 catalog_manager.cc:5484] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "79e41f25066e4e5abbd71e48d545c6bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:01.111188 8821 consensus_queue.cc:237] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:01.113399 9082 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } }
I20240503 02:15:01.126046 8425 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.126956 11317 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.127233 8556 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.129246 11315 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.132556 9212 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.133044 8689 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.133430 9082 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.133370 11315 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.133702 11323 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.134126 11315 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.134594 9212 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } }
I20240503 02:15:01.136320 11317 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:01.136715 8689 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:01.140704 8267 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9b41db0e8b114c99ab1d8734ec8b008f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:01.141531 8425 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:01.144062 8556 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:01.145716 8275 catalog_manager.cc:5484] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf reported cstate change: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New cstate: current_term: 1 leader_uuid: "79e41f25066e4e5abbd71e48d545c6bf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:01.153220 9213 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:01.155334 9082 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:01.162806 8949 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.163924 11317 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.165402 8949 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } }
W20240503 02:15:01.169752 8756 consensus_peers.cc:485] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf -> Peer dabeea74472746538132a520710248ea (127.23.104.202:37759): Couldn't send request to peer dabeea74472746538132a520710248ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b41db0e8b114c99ab1d8734ec8b008f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:01.171591 8949 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:01.180817 9082 consensus_queue.cc:237] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:15:01.183595 8821 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.184170 8425 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.184173 8689 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.184885 8949 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.184789 11332 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.185102 9212 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.185201 8556 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.185487 11129 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.185739 11332 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.186090 11279 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.186383 10997 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.186645 11278 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.204916 10997 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.207012 8949 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.208765 9213 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.213881 8819 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.215255 8281 catalog_manager.cc:5484] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.220113 8689 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.221135 8425 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.229104 9082 consensus_queue.cc:237] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:01.232575 8557 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.232939 8950 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.233484 11332 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.233996 11332 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.235217 9212 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.234546 8557 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.239575 11332 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.239923 8821 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.240403 11332 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.240577 8425 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.241307 10997 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.243578 8690 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.245297 11332 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.245874 10997 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.246656 8557 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.248529 8425 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.249194 8690 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.251564 9212 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.254045 8277 catalog_manager.cc:5484] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.257953 8267 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1a30758cbd1d4845bb9eeb02e1a2942a with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:01.258459 8821 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.274312 8949 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
W20240503 02:15:01.288007 9017 consensus_peers.cc:485] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d -> Peer 4792082063bd4d58bc222c9c95b16335 (127.23.104.201:46877): Couldn't send request to peer 4792082063bd4d58bc222c9c95b16335. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a30758cbd1d4845bb9eeb02e1a2942a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:01.304121 8557 consensus_queue.cc:237] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:15:01.307783 8689 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.307870 9212 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.308377 8425 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.308353 11348 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.308660 11141 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.308921 9082 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.309170 11141 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.309607 10445 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.310539 8949 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.311483 10445 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.314811 8821 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.315946 11141 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.329366 11348 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:01.334798 9082 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:01.335430 8273 catalog_manager.cc:5484] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:01.335518 9212 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:01.347934 8821 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:01.348160 8689 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:01.348414 8557 consensus_queue.cc:237] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:01.350226 8949 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:01.358144 9082 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.358641 11348 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.358745 8821 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.359242 11348 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.359997 8950 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.362314 9213 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.363162 11348 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.364948 8425 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.365458 11141 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.367043 8425 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:01.366679 8689 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.368016 11349 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.373436 8821 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:01.373665 11348 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:01.374465 8689 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:01.378229 11356 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.380206 8269 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 99e42dca2db54de5a9417df20242e2c2 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:01.380614 8950 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:01.383011 8425 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:01.382907 9212 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:01.385004 9083 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:01.389868 8273 catalog_manager.cc:5484] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
W20240503 02:15:01.413782 8492 consensus_peers.cc:485] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 -> Peer dabeea74472746538132a520710248ea (127.23.104.202:37759): Couldn't send request to peer dabeea74472746538132a520710248ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99e42dca2db54de5a9417df20242e2c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:01.422731 9212 consensus_queue.cc:237] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:15:01.425611 8689 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.425608 8821 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.425626 9083 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.426297 11171 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.426468 8949 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.426802 8425 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.427131 11295 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.427513 10510 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.427575 8557 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.428121 11171 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.428493 10510 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.428759 11297 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.433395 11296 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.433931 8425 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.435366 8557 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.438282 9082 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.438637 8689 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.443550 8277 catalog_manager.cc:5484] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.443496 8949 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.450788 9212 consensus_queue.cc:237] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:01.454038 8557 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.454581 11171 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.458120 8690 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.458878 8425 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.459048 11298 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.459673 11295 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.461741 8821 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.462399 8949 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:15:01.462884 9148 consensus_peers.cc:485] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 -> Peer 4792082063bd4d58bc222c9c95b16335 (127.23.104.201:46877): Couldn't send request to peer 4792082063bd4d58bc222c9c95b16335. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00fc93702514499eb32902238c46356d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:01.462951 11296 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.463368 11297 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.464632 9083 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.464228 10510 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.464833 8557 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.465685 8819 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.467912 11296 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.469681 9083 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.474184 8269 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 00fc93702514499eb32902238c46356d with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:01.473826 8952 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.475556 8425 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.476215 8690 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.480680 8819 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.482525 8273 catalog_manager.cc:5484] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.493260 9083 consensus_queue.cc:237] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:15:01.496212 8952 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.496491 8819 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.496793 11129 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.496999 8690 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.497061 11332 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.497444 11278 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.499207 8425 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.499776 11278 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.502862 9212 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.503607 8557 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:01.503662 11129 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.505836 11129 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.512375 11278 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.515926 8952 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.516506 8425 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.516561 8690 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.521796 8276 catalog_manager.cc:5484] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:01.524237 8557 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.536522 9083 consensus_queue.cc:237] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
W20240503 02:15:01.540729 9017 consensus_peers.cc:485] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d -> Peer 4792082063bd4d58bc222c9c95b16335 (127.23.104.201:46877): Couldn't send request to peer 4792082063bd4d58bc222c9c95b16335. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc3f83e9a30047989c4281e049f0a68c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:01.545554 8690 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.546218 8556 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.546926 8952 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.546998 11278 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.547585 11278 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.549201 11129 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.550256 11385 ts_tablet_manager.cc:902] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df: Initiating tablet copy from peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:15:01.550436 8819 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.552899 11332 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.554831 8425 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.555557 11332 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:01.555950 8821 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.556862 11332 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.561540 8556 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.567329 8275 catalog_manager.cc:5484] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.568483 9213 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:01.569551 11278 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:01.569545 8821 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.572403 8267 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fc3f83e9a30047989c4281e049f0a68c with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:01.573381 8690 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.573776 8425 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.575348 8952 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.595268 9212 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:01.601909 9212 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:01.602619 11385 tablet_copy_client.cc:308] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:41237
I20240503 02:15:01.613874 11401 ts_tablet_manager.cc:902] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df: Initiating tablet copy from peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:15:01.631587 11404 ts_tablet_manager.cc:902] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335: Initiating tablet copy from peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:15:01.643595 11403 ts_tablet_manager.cc:902] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea: Initiating tablet copy from peer 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747)
I20240503 02:15:01.643599 9103 tablet_copy_service.cc:140] P c399efcbba924863ac89d156dc07ac7d: Received BeginTabletCopySession request for tablet df68cc6032b14cae9e1cac074976baf8 from peer e512cb0d8f244ac6ab0e7150a0a8e2df ({username='slave'} at 127.23.104.200:52887)
I20240503 02:15:01.643952 9103 tablet_copy_service.cc:161] P c399efcbba924863ac89d156dc07ac7d: Beginning new tablet copy session on tablet df68cc6032b14cae9e1cac074976baf8 from peer e512cb0d8f244ac6ab0e7150a0a8e2df at {username='slave'} at 127.23.104.200:52887: session id = e512cb0d8f244ac6ab0e7150a0a8e2df-df68cc6032b14cae9e1cac074976baf8
I20240503 02:15:01.648927 9103 tablet_copy_source_session.cc:205] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:01.655853 11385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df68cc6032b14cae9e1cac074976baf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:01.685847 11401 tablet_copy_client.cc:308] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.199:32809
I20240503 02:15:01.687207 11403 tablet_copy_client.cc:308] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:34747
I20240503 02:15:01.692355 11385 tablet_copy_client.cc:783] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:01.692672 11385 tablet_copy_client.cc:647] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:01.698033 9234 tablet_copy_service.cc:140] P cb90f34dc37b4b16b2a5b5a4ee012865: Received BeginTabletCopySession request for tablet 879167ff15db41a7b4ef21fa10c44b27 from peer e512cb0d8f244ac6ab0e7150a0a8e2df ({username='slave'} at 127.23.104.200:54679)
I20240503 02:15:01.698267 9234 tablet_copy_service.cc:161] P cb90f34dc37b4b16b2a5b5a4ee012865: Beginning new tablet copy session on tablet 879167ff15db41a7b4ef21fa10c44b27 from peer e512cb0d8f244ac6ab0e7150a0a8e2df at {username='slave'} at 127.23.104.200:54679: session id = e512cb0d8f244ac6ab0e7150a0a8e2df-879167ff15db41a7b4ef21fa10c44b27
I20240503 02:15:01.698601 11404 tablet_copy_client.cc:308] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:41237
I20240503 02:15:01.701753 9234 tablet_copy_source_session.cc:205] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:01.707854 11401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879167ff15db41a7b4ef21fa10c44b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:01.711926 11385 tablet_copy_client.cc:523] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:01.716377 11385 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap starting.
I20240503 02:15:01.718235 8841 tablet_copy_service.cc:140] P 79e41f25066e4e5abbd71e48d545c6bf: Received BeginTabletCopySession request for tablet 9b41db0e8b114c99ab1d8734ec8b008f from peer dabeea74472746538132a520710248ea ({username='slave'} at 127.23.104.202:58123)
I20240503 02:15:01.718461 8841 tablet_copy_service.cc:161] P 79e41f25066e4e5abbd71e48d545c6bf: Beginning new tablet copy session on tablet 9b41db0e8b114c99ab1d8734ec8b008f from peer dabeea74472746538132a520710248ea at {username='slave'} at 127.23.104.202:58123: session id = dabeea74472746538132a520710248ea-9b41db0e8b114c99ab1d8734ec8b008f
I20240503 02:15:01.722254 8841 tablet_copy_source_session.cc:205] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:01.725688 11401 tablet_copy_client.cc:783] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:01.726025 11401 tablet_copy_client.cc:647] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:01.727600 11403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b41db0e8b114c99ab1d8734ec8b008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:01.734027 11401 tablet_copy_client.cc:523] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:01.740959 11403 tablet_copy_client.cc:783] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:01.741263 11403 tablet_copy_client.cc:647] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:01.743523 11403 tablet_copy_client.cc:523] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:01.746083 9103 tablet_copy_service.cc:140] P c399efcbba924863ac89d156dc07ac7d: Received BeginTabletCopySession request for tablet 1a30758cbd1d4845bb9eeb02e1a2942a from peer 4792082063bd4d58bc222c9c95b16335 ({username='slave'} at 127.23.104.201:41501)
I20240503 02:15:01.746291 9103 tablet_copy_service.cc:161] P c399efcbba924863ac89d156dc07ac7d: Beginning new tablet copy session on tablet 1a30758cbd1d4845bb9eeb02e1a2942a from peer 4792082063bd4d58bc222c9c95b16335 at {username='slave'} at 127.23.104.201:41501: session id = 4792082063bd4d58bc222c9c95b16335-1a30758cbd1d4845bb9eeb02e1a2942a
I20240503 02:15:01.747376 11403 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea: Bootstrap starting.
I20240503 02:15:01.750077 9103 tablet_copy_source_session.cc:205] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:01.755396 11401 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap starting.
I20240503 02:15:01.762718 11403 log.cc:824] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea: Log is configured to *not* fsync() on all Append() calls
I20240503 02:15:01.767642 11404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a30758cbd1d4845bb9eeb02e1a2942a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:01.772908 11385 log.cc:824] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df: Log is configured to *not* fsync() on all Append() calls
I20240503 02:15:01.774571 11430 ts_tablet_manager.cc:902] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea: Initiating tablet copy from peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923)
I20240503 02:15:01.811826 11404 tablet_copy_client.cc:783] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:01.812309 11404 tablet_copy_client.cc:647] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:01.814646 11404 tablet_copy_client.cc:523] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:01.821573 11404 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335: Bootstrap starting.
I20240503 02:15:01.824535 11430 tablet_copy_client.cc:308] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:45923
I20240503 02:15:01.852437 11403 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:01.853058 11403 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea: Bootstrap complete.
I20240503 02:15:01.853615 11403 ts_tablet_manager.cc:1367] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea: Time spent bootstrapping tablet: real 0.106s user 0.049s sys 0.029s
I20240503 02:15:01.856483 8578 tablet_copy_service.cc:140] P 2a1893ab239f4163a06bdbf8ce579431: Received BeginTabletCopySession request for tablet 99e42dca2db54de5a9417df20242e2c2 from peer dabeea74472746538132a520710248ea ({username='slave'} at 127.23.104.202:33615)
I20240503 02:15:01.856801 8578 tablet_copy_service.cc:161] P 2a1893ab239f4163a06bdbf8ce579431: Beginning new tablet copy session on tablet 99e42dca2db54de5a9417df20242e2c2 from peer dabeea74472746538132a520710248ea at {username='slave'} at 127.23.104.202:33615: session id = dabeea74472746538132a520710248ea-99e42dca2db54de5a9417df20242e2c2
I20240503 02:15:01.860426 8578 tablet_copy_source_session.cc:205] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:01.860203 11403 raft_consensus.cc:348] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:01.860771 11403 raft_consensus.cc:729] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dabeea74472746538132a520710248ea, State: Initialized, Role: LEARNER
I20240503 02:15:01.861385 11403 consensus_queue.cc:260] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:01.862267 11430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e42dca2db54de5a9417df20242e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:01.864692 11404 log.cc:824] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335: Log is configured to *not* fsync() on all Append() calls
I20240503 02:15:01.868603 11430 tablet_copy_client.cc:783] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:01.868853 11430 tablet_copy_client.cc:647] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:01.871670 11430 tablet_copy_client.cc:523] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:01.876910 11403 ts_tablet_manager.cc:1398] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea: Time spent starting tablet: real 0.023s user 0.013s sys 0.004s
I20240503 02:15:01.882727 8841 tablet_copy_service.cc:342] P 79e41f25066e4e5abbd71e48d545c6bf: Request end of tablet copy session dabeea74472746538132a520710248ea-9b41db0e8b114c99ab1d8734ec8b008f received from {username='slave'} at 127.23.104.202:58123
I20240503 02:15:01.882915 8841 tablet_copy_service.cc:433] P 79e41f25066e4e5abbd71e48d545c6bf: ending tablet copy session dabeea74472746538132a520710248ea-9b41db0e8b114c99ab1d8734ec8b008f on tablet 9b41db0e8b114c99ab1d8734ec8b008f with peer dabeea74472746538132a520710248ea
I20240503 02:15:01.889845 11430 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea: Bootstrap starting.
I20240503 02:15:01.902395 11401 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:01.902943 11401 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap complete.
I20240503 02:15:01.903437 11401 ts_tablet_manager.cc:1367] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent bootstrapping tablet: real 0.148s user 0.050s sys 0.006s
I20240503 02:15:01.906463 11385 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:01.906888 11385 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap complete.
I20240503 02:15:01.907142 11385 ts_tablet_manager.cc:1367] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent bootstrapping tablet: real 0.191s user 0.041s sys 0.023s
I20240503 02:15:01.910364 11401 raft_consensus.cc:348] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:01.910928 11401 raft_consensus.cc:729] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e512cb0d8f244ac6ab0e7150a0a8e2df, State: Initialized, Role: LEARNER
I20240503 02:15:01.911460 11401 consensus_queue.cc:260] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:01.912081 11385 raft_consensus.cc:348] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:01.912664 11385 raft_consensus.cc:729] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e512cb0d8f244ac6ab0e7150a0a8e2df, State: Initialized, Role: LEARNER
I20240503 02:15:01.912873 11385 consensus_queue.cc:260] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:01.917016 11401 ts_tablet_manager.cc:1398] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20240503 02:15:01.920969 9234 tablet_copy_service.cc:342] P cb90f34dc37b4b16b2a5b5a4ee012865: Request end of tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-879167ff15db41a7b4ef21fa10c44b27 received from {username='slave'} at 127.23.104.200:54679
I20240503 02:15:01.921137 9234 tablet_copy_service.cc:433] P cb90f34dc37b4b16b2a5b5a4ee012865: ending tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-879167ff15db41a7b4ef21fa10c44b27 on tablet 879167ff15db41a7b4ef21fa10c44b27 with peer e512cb0d8f244ac6ab0e7150a0a8e2df
I20240503 02:15:01.930714 11385 ts_tablet_manager.cc:1398] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent starting tablet: real 0.023s user 0.009s sys 0.008s
I20240503 02:15:01.931821 9103 tablet_copy_service.cc:342] P c399efcbba924863ac89d156dc07ac7d: Request end of tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-df68cc6032b14cae9e1cac074976baf8 received from {username='slave'} at 127.23.104.200:52887
I20240503 02:15:01.931959 9103 tablet_copy_service.cc:433] P c399efcbba924863ac89d156dc07ac7d: ending tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-df68cc6032b14cae9e1cac074976baf8 on tablet df68cc6032b14cae9e1cac074976baf8 with peer e512cb0d8f244ac6ab0e7150a0a8e2df
I20240503 02:15:01.946501 10816 raft_consensus.cc:1203] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:01.969786 11404 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:01.970330 11404 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335: Bootstrap complete.
I20240503 02:15:01.970824 11404 ts_tablet_manager.cc:1367] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335: Time spent bootstrapping tablet: real 0.150s user 0.066s sys 0.014s
I20240503 02:15:01.977437 11404 raft_consensus.cc:348] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:01.978001 11404 raft_consensus.cc:729] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4792082063bd4d58bc222c9c95b16335, State: Initialized, Role: LEARNER
I20240503 02:15:01.978581 11404 consensus_queue.cc:260] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:01.982867 11404 ts_tablet_manager.cc:1398] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335: Time spent starting tablet: real 0.012s user 0.005s sys 0.008s
I20240503 02:15:01.983845 9103 tablet_copy_service.cc:342] P c399efcbba924863ac89d156dc07ac7d: Request end of tablet copy session 4792082063bd4d58bc222c9c95b16335-1a30758cbd1d4845bb9eeb02e1a2942a received from {username='slave'} at 127.23.104.201:41501
I20240503 02:15:01.983973 9103 tablet_copy_service.cc:433] P c399efcbba924863ac89d156dc07ac7d: ending tablet copy session 4792082063bd4d58bc222c9c95b16335-1a30758cbd1d4845bb9eeb02e1a2942a on tablet 1a30758cbd1d4845bb9eeb02e1a2942a with peer 4792082063bd4d58bc222c9c95b16335
I20240503 02:15:02.006649 11430 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:02.007081 11430 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea: Bootstrap complete.
I20240503 02:15:02.007383 11430 ts_tablet_manager.cc:1367] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea: Time spent bootstrapping tablet: real 0.123s user 0.039s sys 0.018s
I20240503 02:15:02.008210 11430 raft_consensus.cc:348] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:02.008442 11430 raft_consensus.cc:729] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dabeea74472746538132a520710248ea, State: Initialized, Role: LEARNER
I20240503 02:15:02.008643 11430 consensus_queue.cc:260] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:02.009114 11430 ts_tablet_manager.cc:1398] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:15:02.009812 8578 tablet_copy_service.cc:342] P 2a1893ab239f4163a06bdbf8ce579431: Request end of tablet copy session dabeea74472746538132a520710248ea-99e42dca2db54de5a9417df20242e2c2 received from {username='slave'} at 127.23.104.202:33615
I20240503 02:15:02.009958 8578 tablet_copy_service.cc:433] P 2a1893ab239f4163a06bdbf8ce579431: ending tablet copy session dabeea74472746538132a520710248ea-99e42dca2db54de5a9417df20242e2c2 on tablet 99e42dca2db54de5a9417df20242e2c2 with peer dabeea74472746538132a520710248ea
I20240503 02:15:02.062208 10816 raft_consensus.cc:1203] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:02.079205 11404 ts_tablet_manager.cc:902] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335: Initiating tablet copy from peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:15:02.080459 11404 tablet_copy_client.cc:308] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.199:32809
I20240503 02:15:02.085842 9234 tablet_copy_service.cc:140] P cb90f34dc37b4b16b2a5b5a4ee012865: Received BeginTabletCopySession request for tablet 00fc93702514499eb32902238c46356d from peer 4792082063bd4d58bc222c9c95b16335 ({username='slave'} at 127.23.104.201:35255)
I20240503 02:15:02.085988 9234 tablet_copy_service.cc:161] P cb90f34dc37b4b16b2a5b5a4ee012865: Beginning new tablet copy session on tablet 00fc93702514499eb32902238c46356d from peer 4792082063bd4d58bc222c9c95b16335 at {username='slave'} at 127.23.104.201:35255: session id = 4792082063bd4d58bc222c9c95b16335-00fc93702514499eb32902238c46356d
I20240503 02:15:02.086547 11463 ts_tablet_manager.cc:902] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335: Initiating tablet copy from peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:15:02.088322 9234 tablet_copy_source_session.cc:205] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:02.089529 11404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fc93702514499eb32902238c46356d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:02.092160 11463 tablet_copy_client.cc:308] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:41237
I20240503 02:15:02.092826 9103 tablet_copy_service.cc:140] P c399efcbba924863ac89d156dc07ac7d: Received BeginTabletCopySession request for tablet fc3f83e9a30047989c4281e049f0a68c from peer 4792082063bd4d58bc222c9c95b16335 ({username='slave'} at 127.23.104.201:41501)
I20240503 02:15:02.092979 9103 tablet_copy_service.cc:161] P c399efcbba924863ac89d156dc07ac7d: Beginning new tablet copy session on tablet fc3f83e9a30047989c4281e049f0a68c from peer 4792082063bd4d58bc222c9c95b16335 at {username='slave'} at 127.23.104.201:41501: session id = 4792082063bd4d58bc222c9c95b16335-fc3f83e9a30047989c4281e049f0a68c
I20240503 02:15:02.094877 11404 tablet_copy_client.cc:783] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:02.095106 11404 tablet_copy_client.cc:647] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:02.095921 9103 tablet_copy_source_session.cc:205] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:02.096700 11404 tablet_copy_client.cc:523] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:02.097055 11463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3f83e9a30047989c4281e049f0a68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:02.099745 11404 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335: Bootstrap starting.
I20240503 02:15:02.102460 11463 tablet_copy_client.cc:783] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:02.102721 11463 tablet_copy_client.cc:647] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:02.104292 11463 tablet_copy_client.cc:523] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:02.106693 11463 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335: Bootstrap starting.
I20240503 02:15:02.134104 11404 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:02.134536 11404 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335: Bootstrap complete.
I20240503 02:15:02.134788 11404 ts_tablet_manager.cc:1367] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335: Time spent bootstrapping tablet: real 0.035s user 0.033s sys 0.000s
I20240503 02:15:02.135675 11404 raft_consensus.cc:348] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:02.135980 11404 raft_consensus.cc:729] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4792082063bd4d58bc222c9c95b16335, State: Initialized, Role: LEARNER
I20240503 02:15:02.136201 11404 consensus_queue.cc:260] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:02.137418 11404 ts_tablet_manager.cc:1398] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240503 02:15:02.139039 9234 tablet_copy_service.cc:342] P cb90f34dc37b4b16b2a5b5a4ee012865: Request end of tablet copy session 4792082063bd4d58bc222c9c95b16335-00fc93702514499eb32902238c46356d received from {username='slave'} at 127.23.104.201:35255
I20240503 02:15:02.139163 9234 tablet_copy_service.cc:433] P cb90f34dc37b4b16b2a5b5a4ee012865: ending tablet copy session 4792082063bd4d58bc222c9c95b16335-00fc93702514499eb32902238c46356d on tablet 00fc93702514499eb32902238c46356d with peer 4792082063bd4d58bc222c9c95b16335
I20240503 02:15:02.148275 11463 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:02.148612 11463 tablet_bootstrap.cc:492] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335: Bootstrap complete.
I20240503 02:15:02.148813 11463 ts_tablet_manager.cc:1367] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335: Time spent bootstrapping tablet: real 0.042s user 0.039s sys 0.003s
I20240503 02:15:02.149340 11463 raft_consensus.cc:348] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:02.149556 11463 raft_consensus.cc:729] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4792082063bd4d58bc222c9c95b16335, State: Initialized, Role: LEARNER
I20240503 02:15:02.149755 11463 consensus_queue.cc:260] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:02.150249 11463 ts_tablet_manager.cc:1398] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:15:02.150898 9103 tablet_copy_service.cc:342] P c399efcbba924863ac89d156dc07ac7d: Request end of tablet copy session 4792082063bd4d58bc222c9c95b16335-fc3f83e9a30047989c4281e049f0a68c received from {username='slave'} at 127.23.104.201:41501
I20240503 02:15:02.151047 9103 tablet_copy_service.cc:433] P c399efcbba924863ac89d156dc07ac7d: ending tablet copy session 4792082063bd4d58bc222c9c95b16335-fc3f83e9a30047989c4281e049f0a68c on tablet fc3f83e9a30047989c4281e049f0a68c with peer 4792082063bd4d58bc222c9c95b16335
I20240503 02:15:02.172502 11079 raft_consensus.cc:1203] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:02.195178 11079 raft_consensus.cc:1203] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:02.240589 10947 raft_consensus.cc:1203] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:02.335428 11279 raft_consensus.cc:1050] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d: attempting to promote NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df to VOTER
I20240503 02:15:02.337111 11279 consensus_queue.cc:237] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:02.340797 8690 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.341003 8556 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.341461 11278 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.341802 11332 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.342811 10816 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.345330 11507 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.346683 8424 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.346933 9212 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.347298 11332 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.347709 11332 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.348731 8952 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.349318 11278 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.349903 8821 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.351490 11278 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.377780 11507 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.378455 8424 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.380999 9213 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.384111 8690 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.384946 8821 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.386574 8279 catalog_manager.cc:5484] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d reported cstate change: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:02.389492 8952 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.395079 8556 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.399950 10816 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.418409 9083 consensus_queue.cc:237] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:02.422046 10816 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.422255 8424 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.422369 8690 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.423022 11507 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.423338 8556 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.423707 8821 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.423921 11507 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.424223 11279 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.425032 11507 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.425598 11332 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.426635 9212 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.427253 11332 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.428109 11332 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.428892 8690 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.432125 8556 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.433327 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df68cc6032b14cae9e1cac074976baf8 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:02.437975 10816 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.435778 8277 catalog_manager.cc:5484] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 reported cstate change: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.439229 8424 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.441951 8821 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.443285 9213 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.452661 8931 tablet_service.cc:1514] Processing DeleteTablet for tablet df68cc6032b14cae9e1cac074976baf8 with delete_type TABLET_DATA_TOMBSTONED (TS 99be4ff1114d4aaabcc537604fce54b3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52056
I20240503 02:15:02.453615 11519 tablet_replica.cc:330] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3: stopping tablet replica
I20240503 02:15:02.454293 11519 raft_consensus.cc:2229] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:02.454950 11519 raft_consensus.cc:2258] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:02.456916 11519 ts_tablet_manager.cc:1875] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:02.462613 11519 ts_tablet_manager.cc:1888] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:02.462765 11519 log.cc:1196] T df68cc6032b14cae9e1cac074976baf8 P 99be4ff1114d4aaabcc537604fce54b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/wals/df68cc6032b14cae9e1cac074976baf8
I20240503 02:15:02.463631 8268 catalog_manager.cc:4830] TS 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): tablet df68cc6032b14cae9e1cac074976baf8 (table rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]) successfully deleted
I20240503 02:15:02.477613 11296 raft_consensus.cc:1050] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865: attempting to promote NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df to VOTER
I20240503 02:15:02.478631 11296 consensus_queue.cc:237] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:02.494963 8556 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.496083 10816 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.496621 11295 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.496941 11521 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.498400 8690 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.499141 11295 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.499445 8424 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.500329 10510 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.500746 9083 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.501120 8819 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.501292 8952 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.502672 10510 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.503093 10510 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.503862 10510 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.537403 10510 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.540532 8821 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.544682 8690 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.547190 8282 catalog_manager.cc:5484] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf reported cstate change: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.546522 8952 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.551400 10816 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.551442 8424 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.560187 8556 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.560688 9213 consensus_queue.cc:237] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:02.562516 11470 raft_consensus.cc:1050] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf: attempting to promote NON_VOTER dabeea74472746538132a520710248ea to VOTER
I20240503 02:15:02.563745 11470 consensus_queue.cc:237] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:02.565265 10947 raft_consensus.cc:1203] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:02.565809 10816 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.566917 10510 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.568437 8424 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.569628 10510 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.571614 8557 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.572489 10510 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.573621 8690 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.576408 8949 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.578263 9081 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.577896 9083 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.578502 11079 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea [term 1 LEARNER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.579208 11323 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.581023 11317 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.581756 11317 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.584746 9082 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.590554 8952 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.591286 11531 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.596802 8424 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.596832 8556 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.597733 8690 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.597640 11323 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.598325 11323 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.598805 11533 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.604291 11531 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.605820 8557 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.611709 8269 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 879167ff15db41a7b4ef21fa10c44b27 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:02.613157 10816 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.619705 8274 catalog_manager.cc:5484] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df reported cstate change: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.620497 11295 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.621115 11295 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.621675 8950 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.623442 9212 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.624933 8801 tablet_service.cc:1514] Processing DeleteTablet for tablet 879167ff15db41a7b4ef21fa10c44b27 with delete_type TABLET_DATA_TOMBSTONED (TS 79e41f25066e4e5abbd71e48d545c6bf not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38550
I20240503 02:15:02.639400 11546 tablet_replica.cc:330] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: stopping tablet replica
I20240503 02:15:02.640247 11546 raft_consensus.cc:2229] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:02.640926 11546 raft_consensus.cc:2258] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:02.642959 11546 ts_tablet_manager.cc:1875] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:02.643579 10947 raft_consensus.cc:1203] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:02.643507 8424 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.645082 9082 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.645489 8689 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:02.654372 11533 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.670583 11546 ts_tablet_manager.cc:1888] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:02.672766 11546 log.cc:1196] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/879167ff15db41a7b4ef21fa10c44b27
I20240503 02:15:02.674037 8267 catalog_manager.cc:4830] TS 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747): tablet 879167ff15db41a7b4ef21fa10c44b27 (table rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]) successfully deleted
I20240503 02:15:02.677968 11279 raft_consensus.cc:1050] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d: attempting to promote NON_VOTER 4792082063bd4d58bc222c9c95b16335 to VOTER
I20240503 02:15:02.679576 11279 consensus_queue.cc:237] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:02.769096 11356 raft_consensus.cc:1050] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431: attempting to promote NON_VOTER dabeea74472746538132a520710248ea to VOTER
I20240503 02:15:02.770870 11356 consensus_queue.cc:237] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:02.790544 10947 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.791565 11510 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.793637 9213 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.795373 8821 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.795737 11510 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.796100 11507 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.798084 8950 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.797564 11317 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:02.798837 11332 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.801070 8952 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:02.817126 8274 catalog_manager.cc:5484] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79e41f25066e4e5abbd71e48d545c6bf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:02.820935 8689 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.821792 11510 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.824466 8557 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.825207 11510 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.834362 8424 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.835315 11510 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 02:15:02.849907 11507 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:02.851014 9213 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:02.857622 9214 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.858767 11563 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.859283 8821 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:02.861727 10947 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:02.865494 8275 catalog_manager.cc:5484] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d reported cstate change: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:15:02.868512 8950 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:02.875066 11078 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:02.888389 8424 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:02.888404 8689 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:02.891834 9082 consensus_queue.cc:237] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:02.896723 8557 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:02.899721 9213 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.900154 8821 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.900305 11507 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.900700 11510 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.902810 9213 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:02.903851 10947 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.905210 11507 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.909262 9213 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:02.916733 8690 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.917415 11510 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.916833 11279 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:02.921772 8689 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:02.929391 8274 catalog_manager.cc:5484] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:02.944954 11079 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.945240 8425 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:02.946126 8949 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.946372 11510 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:02.947944 11348 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.948630 11348 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.954349 8821 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.954525 9082 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.958221 11563 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.958516 11562 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.959338 8556 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:02.964668 8425 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:02.969569 11562 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:02.976151 8424 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:02.978986 8931 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a30758cbd1d4845bb9eeb02e1a2942a with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 6 (current committed config index is 7)) from {username='slave'} at 127.0.0.1:52056
I20240503 02:15:02.983990 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1a30758cbd1d4845bb9eeb02e1a2942a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:02.987135 10947 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:02.987457 8821 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:02.951720 8690 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:03.006739 11571 tablet_replica.cc:330] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: stopping tablet replica
I20240503 02:15:03.006940 11571 raft_consensus.cc:2229] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:03.007220 11571 raft_consensus.cc:2258] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:03.008487 11356 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.009299 11571 ts_tablet_manager.cc:1875] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:03.009670 9082 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.015861 11079 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.017439 8558 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.018332 11279 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.020434 8558 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.022583 9213 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.022854 8275 catalog_manager.cc:5484] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea reported cstate change: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.028878 11560 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:03.030833 8950 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.035740 11571 ts_tablet_manager.cc:1888] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:03.045220 11571 log.cc:1196] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/wals/1a30758cbd1d4845bb9eeb02e1a2942a
I20240503 02:15:03.046267 8268 catalog_manager.cc:4830] TS 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): tablet 1a30758cbd1d4845bb9eeb02e1a2942a (table rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]) successfully deleted
I20240503 02:15:03.058701 8425 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.063058 8819 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.073567 9082 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.077193 8687 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.077615 8424 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.072741 8690 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.106056 8821 consensus_queue.cc:237] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:03.111078 11079 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.111297 9082 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.111407 8950 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.112291 11574 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.113134 11317 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.113373 11574 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.117561 11574 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.118955 8950 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.122185 8557 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.123018 8423 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.123659 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b41db0e8b114c99ab1d8734ec8b008f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:03.124102 11323 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.125756 11470 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.128845 8423 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.127241 8557 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.132138 8690 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.134979 11539 consensus_queue.cc:1037] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 1.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240503 02:15:03.138084 8689 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.145067 8282 catalog_manager.cc:5484] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index 6 to 7, VOTER cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199) evicted. New cstate: current_term: 1 leader_uuid: "79e41f25066e4e5abbd71e48d545c6bf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.147136 11510 raft_consensus.cc:1050] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d: attempting to promote NON_VOTER 4792082063bd4d58bc222c9c95b16335 to VOTER
I20240503 02:15:03.149117 11510 consensus_queue.cc:237] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:03.158830 11079 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.164454 9082 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.178372 9193 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b41db0e8b114c99ab1d8734ec8b008f with delete_type TABLET_DATA_TOMBSTONED (TS cb90f34dc37b4b16b2a5b5a4ee012865 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55178
I20240503 02:15:03.198963 11580 tablet_replica.cc:330] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865: stopping tablet replica
I20240503 02:15:03.199555 11580 raft_consensus.cc:2229] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:03.200229 11580 raft_consensus.cc:2258] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:03.201843 11580 ts_tablet_manager.cc:1875] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:03.212527 10947 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:03.213452 11279 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:03.213583 8950 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:03.214810 11332 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:03.215050 8557 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:03.215600 8423 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:03.215610 8689 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:03.216570 11507 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:03.219287 8819 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:03.219689 11279 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:03.221256 11332 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 02:15:03.222007 11332 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:03.225926 9212 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:03.226693 11279 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:03.228449 11580 ts_tablet_manager.cc:1888] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:03.228678 11580 log.cc:1196] T 9b41db0e8b114c99ab1d8734ec8b008f P cb90f34dc37b4b16b2a5b5a4ee012865: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-6/wal/wals/9b41db0e8b114c99ab1d8734ec8b008f
I20240503 02:15:03.237442 8269 catalog_manager.cc:4830] TS cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809): tablet 9b41db0e8b114c99ab1d8734ec8b008f (table rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]) successfully deleted
I20240503 02:15:03.242049 10510 raft_consensus.cc:1050] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865: attempting to promote NON_VOTER 4792082063bd4d58bc222c9c95b16335 to VOTER
I20240503 02:15:03.243222 10510 consensus_queue.cc:237] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:03.251776 10947 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:03.252424 11296 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:03.254452 8821 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:03.255381 9082 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:03.256122 8423 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:03.256227 8949 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:03.257342 11521 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:03.257849 11521 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:03.258332 11521 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:03.258786 11521 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:03.263078 11279 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.264391 8690 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.271224 8423 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.271487 8275 catalog_manager.cc:5484] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 reported cstate change: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.274653 8557 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:03.275507 10510 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:03.276319 10947 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.283108 8558 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.292443 8689 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:03.297658 8949 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
W20240503 02:15:03.302214 9148 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:41237, user_credentials={real_user=slave}} blocked reactor thread for 45008.6us
I20240503 02:15:03.314718 11521 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:03.335994 11521 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.347833 8949 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.348476 8556 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.358511 8275 catalog_manager.cc:5484] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.363430 8423 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.364764 9212 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.369534 10946 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.373723 9081 consensus_queue.cc:237] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:03.383527 8425 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.383534 8690 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.384266 11510 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.384574 8952 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.384996 11510 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.385249 11587 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.386152 9212 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.387122 10946 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.388479 8556 raft_consensus.cc:1261] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.389144 11279 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.389612 11279 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.392536 11510 consensus_queue.cc:1035] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.393086 11510 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.395736 8557 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.396781 8425 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.395413 8819 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.400588 9212 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.404331 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fc3f83e9a30047989c4281e049f0a68c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:03.405500 8282 catalog_manager.cc:5484] T fc3f83e9a30047989c4281e049f0a68c P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.405368 8690 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.406625 10946 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.412097 8950 raft_consensus.cc:2932] T fc3f83e9a30047989c4281e049f0a68c P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.414930 8801 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3f83e9a30047989c4281e049f0a68c with delete_type TABLET_DATA_TOMBSTONED (TS 79e41f25066e4e5abbd71e48d545c6bf not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38550
I20240503 02:15:03.418560 8820 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.418637 9082 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.421239 11611 tablet_replica.cc:330] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf: stopping tablet replica
I20240503 02:15:03.424083 11611 raft_consensus.cc:2229] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:03.424441 11611 raft_consensus.cc:2258] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:03.425459 11611 ts_tablet_manager.cc:1875] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:03.427129 8690 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.432878 9213 consensus_queue.cc:237] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:03.434789 11611 ts_tablet_manager.cc:1888] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:03.434935 11611 log.cc:1196] T fc3f83e9a30047989c4281e049f0a68c P 79e41f25066e4e5abbd71e48d545c6bf: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/fc3f83e9a30047989c4281e049f0a68c
I20240503 02:15:03.435931 8556 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.436489 11521 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.438436 8819 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.439278 10946 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.439935 8267 catalog_manager.cc:4830] TS 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747): tablet fc3f83e9a30047989c4281e049f0a68c (table rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]) successfully deleted
I20240503 02:15:03.441570 9082 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.442718 11598 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.443081 8425 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.443385 8690 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.443341 11598 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240503 02:15:03.444214 11521 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.446115 11521 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.446352 11598 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.448701 11521 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.449447 9082 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.450788 8425 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.452775 8269 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00fc93702514499eb32902238c46356d with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:03.454018 8819 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.454020 8556 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.459611 8274 catalog_manager.cc:5484] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.465546 8931 tablet_service.cc:1514] Processing DeleteTablet for tablet 00fc93702514499eb32902238c46356d with delete_type TABLET_DATA_TOMBSTONED (TS 99be4ff1114d4aaabcc537604fce54b3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52056
I20240503 02:15:03.465929 11571 tablet_replica.cc:330] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: stopping tablet replica
I20240503 02:15:03.466116 11571 raft_consensus.cc:2229] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:03.466586 11571 raft_consensus.cc:2258] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:03.466886 10946 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.468050 11571 ts_tablet_manager.cc:1875] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:03.465910 8690 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:03.480804 11571 ts_tablet_manager.cc:1888] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:03.480990 11571 log.cc:1196] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/wals/00fc93702514499eb32902238c46356d
I20240503 02:15:03.481820 8268 catalog_manager.cc:4830] TS 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): tablet 00fc93702514499eb32902238c46356d (table rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]) successfully deleted
I20240503 02:15:03.648396 8556 consensus_queue.cc:237] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:03.650820 9212 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.650956 11079 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.651432 11356 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.651597 8690 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.651664 8425 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.651722 9082 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.652379 11356 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.652494 8950 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:03.652647 11628 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.653000 11629 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.653321 11561 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.653985 11561 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:03.657620 11628 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.658335 8425 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.658792 11079 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.661762 8269 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99e42dca2db54de5a9417df20242e2c2 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:03.663709 8950 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.663843 9212 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.665876 8282 catalog_manager.cc:5484] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:15:03.667383 9082 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.669934 8690 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:03.672976 8801 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e42dca2db54de5a9417df20242e2c2 with delete_type TABLET_DATA_TOMBSTONED (TS 79e41f25066e4e5abbd71e48d545c6bf not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38550
I20240503 02:15:03.673426 11611 tablet_replica.cc:330] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf: stopping tablet replica
I20240503 02:15:03.673622 11611 raft_consensus.cc:2229] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:03.673893 11611 raft_consensus.cc:2258] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:03.677605 11611 ts_tablet_manager.cc:1875] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:03.682281 11611 ts_tablet_manager.cc:1888] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:03.682377 11611 log.cc:1196] T 99e42dca2db54de5a9417df20242e2c2 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/99e42dca2db54de5a9417df20242e2c2
I20240503 02:15:03.683024 8267 catalog_manager.cc:4830] TS 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747): tablet 99e42dca2db54de5a9417df20242e2c2 (table rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]) successfully deleted
I20240503 02:15:03.725697 8556 consensus_queue.cc:237] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:15:03.729727 9212 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.730197 8950 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.730324 11561 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.732261 9082 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.733220 8819 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.734246 8690 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.735033 8425 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.736202 11630 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.736501 11629 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.737066 11630 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.737318 11629 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.743916 11628 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.753068 11637 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:03.754045 8819 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:03.754240 8950 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:03.758320 8690 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:03.759275 9212 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:03.760669 8273 catalog_manager.cc:5484] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } }
I20240503 02:15:03.767645 8556 consensus_queue.cc:237] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:03.771199 8950 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.771874 11356 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:15:03.772819 8492 consensus_peers.cc:485] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 -> Peer dabeea74472746538132a520710248ea (127.23.104.202:37759): Couldn't send request to peer dabeea74472746538132a520710248ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 493e2b13e19241fa8fa48c367986caaf. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:03.775254 8690 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.778199 8424 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.778769 9081 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.779296 11630 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.779501 11628 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:03.780565 11356 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.782002 9212 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.782999 9082 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:03.783932 11561 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:03.784883 8425 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:03.788022 8819 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.787396 11561 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:03.788501 11630 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:03.789505 8690 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:03.791556 8820 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:03.792543 8425 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:03.799440 8275 catalog_manager.cc:5484] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:03.800060 9082 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:03.802654 8269 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 493e2b13e19241fa8fa48c367986caaf with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:03.803507 8950 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:03.814339 9212 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:03.825160 9212 consensus_queue.cc:237] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:15:03.828085 8950 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.828167 8820 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.828248 9082 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.828620 8690 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.828720 10510 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.829069 8556 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.829531 8425 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.829581 11521 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.829864 11544 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.830096 11598 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.830813 11544 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.831135 10510 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.844384 11598 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:03.847467 8950 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:03.848387 8424 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:03.849200 8690 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:03.850692 8556 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:03.852721 8273 catalog_manager.cc:5484] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } }
I20240503 02:15:03.854326 8820 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:03.860386 9082 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:03.863126 9212 consensus_queue.cc:237] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:03.867941 8949 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:15:03.867977 9150 consensus_peers.cc:485] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 -> Peer e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200:45027): Couldn't send request to peer e512cb0d8f244ac6ab0e7150a0a8e2df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac90f5b4e0fc43e99b3f76fe92610153. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:03.868491 11598 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:03.869689 9082 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.869804 8424 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.870100 8689 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.870887 8820 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.873574 11544 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:03.873844 11598 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:03.874043 11521 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:03.874203 11296 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:03.876206 8556 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.876817 11521 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:03.879633 11598 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:03.881263 8949 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:03.884764 8269 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ac90f5b4e0fc43e99b3f76fe92610153 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:03.887461 8689 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:03.891921 8424 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:03.892308 9082 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:03.892612 8273 catalog_manager.cc:5484] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:03.897946 8820 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:03.907593 8949 consensus_queue.cc:237] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:15:03.908581 8556 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:03.911568 8820 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.911600 9082 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.912401 11363 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.912724 11224 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.914125 8690 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.914985 11363 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.925678 8424 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.925859 9212 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.926054 8556 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:03.926355 11363 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.928870 11363 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.929291 11668 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.929324 11224 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:03.930745 8820 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:03.934782 8689 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:03.936093 8275 catalog_manager.cc:5484] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:03.941924 9082 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:03.948462 8949 consensus_queue.cc:237] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:03.957087 8424 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.958004 11664 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:03.959597 8424 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:03.964830 8820 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.964896 8689 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.965500 11668 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:03.965986 11668 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:03.970568 8556 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:03.973712 9212 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:03.973657 11664 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:03.978204 8425 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:03.978924 9081 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.981128 9213 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.983587 8268 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:03.983806 11224 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:03.984324 11224 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:03.985332 8689 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:03.986143 8820 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:03.987673 8557 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:03.989375 8276 catalog_manager.cc:5484] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:03.997885 9213 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:04.001415 9082 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:04.011246 11224 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:15:04.013231 8888 consensus_peers.cc:485] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 -> Peer e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200:45027): Couldn't send request to peer e512cb0d8f244ac6ab0e7150a0a8e2df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c28f0e30cfe46fda64dd2b0fd7847c6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:04.015136 8557 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:04.041501 8557 consensus_queue.cc:237] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:15:04.044027 8820 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.044090 8949 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.044469 9082 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.044893 9213 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.045634 11628 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.045962 11629 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.046228 8689 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.046788 8425 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.046907 11561 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.047475 11561 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.047930 11561 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.049062 11356 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.055577 11629 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.056836 9082 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.059770 8949 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.060333 9213 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.062467 8425 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.063884 8820 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.064863 8274 catalog_manager.cc:5484] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.065243 8689 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.078402 8557 consensus_queue.cc:237] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:04.083046 8425 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.083685 8949 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.084051 8689 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.084484 8820 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.084417 11356 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.085467 11628 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.085737 11356 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.086297 11561 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.087561 9082 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.088131 11628 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.089291 11561 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.091964 8269 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7d1f333099d942ae897224a1ee1538a6 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:04.092981 8689 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
W20240503 02:15:04.093731 8492 consensus_peers.cc:485] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 -> Peer dabeea74472746538132a520710248ea (127.23.104.202:37759): Couldn't send request to peer dabeea74472746538132a520710248ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d1f333099d942ae897224a1ee1538a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:04.093432 8820 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.094697 9213 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.096211 11629 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.097816 9213 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.098685 8425 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.100370 8282 catalog_manager.cc:5484] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 reported cstate change: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.104383 8949 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.103957 9082 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.263567 11718 ts_tablet_manager.cc:902] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea: Initiating tablet copy from peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923)
I20240503 02:15:04.274859 11718 tablet_copy_client.cc:308] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:45923
I20240503 02:15:04.275508 8578 tablet_copy_service.cc:140] P 2a1893ab239f4163a06bdbf8ce579431: Received BeginTabletCopySession request for tablet 493e2b13e19241fa8fa48c367986caaf from peer dabeea74472746538132a520710248ea ({username='slave'} at 127.23.104.202:33615)
I20240503 02:15:04.275658 8578 tablet_copy_service.cc:161] P 2a1893ab239f4163a06bdbf8ce579431: Beginning new tablet copy session on tablet 493e2b13e19241fa8fa48c367986caaf from peer dabeea74472746538132a520710248ea at {username='slave'} at 127.23.104.202:33615: session id = dabeea74472746538132a520710248ea-493e2b13e19241fa8fa48c367986caaf
I20240503 02:15:04.277640 8578 tablet_copy_source_session.cc:205] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:04.278740 11718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493e2b13e19241fa8fa48c367986caaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:04.283685 11718 tablet_copy_client.cc:783] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:04.283936 11718 tablet_copy_client.cc:647] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:04.285838 11718 tablet_copy_client.cc:523] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:04.288112 11718 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea: Bootstrap starting.
I20240503 02:15:04.317488 11718 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:04.317919 11718 tablet_bootstrap.cc:492] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea: Bootstrap complete.
I20240503 02:15:04.318141 11718 ts_tablet_manager.cc:1367] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.004s
I20240503 02:15:04.318717 11718 raft_consensus.cc:348] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:04.319025 11718 raft_consensus.cc:729] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dabeea74472746538132a520710248ea, State: Initialized, Role: LEARNER
I20240503 02:15:04.319242 11718 consensus_queue.cc:260] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:04.320314 11718 ts_tablet_manager.cc:1398] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:15:04.321065 8578 tablet_copy_service.cc:342] P 2a1893ab239f4163a06bdbf8ce579431: Request end of tablet copy session dabeea74472746538132a520710248ea-493e2b13e19241fa8fa48c367986caaf received from {username='slave'} at 127.23.104.202:33615
I20240503 02:15:04.321189 8578 tablet_copy_service.cc:433] P 2a1893ab239f4163a06bdbf8ce579431: ending tablet copy session dabeea74472746538132a520710248ea-493e2b13e19241fa8fa48c367986caaf on tablet 493e2b13e19241fa8fa48c367986caaf with peer dabeea74472746538132a520710248ea
I20240503 02:15:04.389338 8949 consensus_queue.cc:237] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:15:04.391891 8820 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.391953 8689 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.391953 9082 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.392413 11363 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.392994 8556 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.393003 11668 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.393414 9213 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.393483 11363 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.393973 11363 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.394515 11363 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.398072 8425 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.398831 11676 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.409263 11676 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.412097 8557 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.413112 9213 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.417330 9082 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.417315 8425 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.420339 8820 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.423399 8274 catalog_manager.cc:5484] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.432637 11737 ts_tablet_manager.cc:902] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df: Initiating tablet copy from peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:15:04.433084 8689 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.433952 8949 consensus_queue.cc:237] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:04.443399 11737 tablet_copy_client.cc:308] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.199:32809
I20240503 02:15:04.444172 9234 tablet_copy_service.cc:140] P cb90f34dc37b4b16b2a5b5a4ee012865: Received BeginTabletCopySession request for tablet ac90f5b4e0fc43e99b3f76fe92610153 from peer e512cb0d8f244ac6ab0e7150a0a8e2df ({username='slave'} at 127.23.104.200:54679)
I20240503 02:15:04.444332 9234 tablet_copy_service.cc:161] P cb90f34dc37b4b16b2a5b5a4ee012865: Beginning new tablet copy session on tablet ac90f5b4e0fc43e99b3f76fe92610153 from peer e512cb0d8f244ac6ab0e7150a0a8e2df at {username='slave'} at 127.23.104.200:54679: session id = e512cb0d8f244ac6ab0e7150a0a8e2df-ac90f5b4e0fc43e99b3f76fe92610153
I20240503 02:15:04.444762 8425 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.447168 9234 tablet_copy_source_session.cc:205] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:04.448159 11668 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.449749 11737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac90f5b4e0fc43e99b3f76fe92610153. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:04.454398 8557 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.457001 9213 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.457104 8689 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.457652 11224 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.458220 8820 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.458287 11676 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.458796 11676 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.459297 11363 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.463225 11737 tablet_copy_client.cc:783] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:04.463591 11737 tablet_copy_client.cc:647] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:04.464128 11224 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.464517 8557 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.467887 11737 tablet_copy_client.cc:523] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:04.471062 8689 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.471751 8820 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.472126 8282 catalog_manager.cc:5484] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.474421 11737 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap starting.
I20240503 02:15:04.475685 9082 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.480913 11668 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.482049 8689 consensus_queue.cc:237] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:15:04.484067 9082 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.488704 8268 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 64c52f9977ae4eb4ac803be99b6426c2 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:04.491210 8425 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.498001 9213 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
W20240503 02:15:04.514657 8887 consensus_peers.cc:485] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 -> Peer dabeea74472746538132a520710248ea (127.23.104.202:37759): Couldn't send request to peer dabeea74472746538132a520710248ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64c52f9977ae4eb4ac803be99b6426c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:04.519451 11749 ts_tablet_manager.cc:902] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Initiating tablet copy from peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403)
I20240503 02:15:04.529704 8820 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.530665 10591 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.531469 8949 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.536406 11753 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.538956 9082 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.539922 10591 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.548904 9213 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.549129 8425 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.549582 11754 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.549925 11753 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.550040 8557 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:04.550861 11753 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 02:15:04.553491 10591 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.559134 8425 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.562840 9082 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.568202 8282 catalog_manager.cc:5484] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.569134 11749 tablet_copy_client.cc:308] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.197:45403
I20240503 02:15:04.570950 8949 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.573390 8820 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.596750 8689 consensus_queue.cc:237] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:04.597194 8556 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.601008 8425 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.602718 11753 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.603756 8556 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.605007 9213 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.606679 10591 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.609678 10591 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:04.611279 9212 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:04.620054 11718 ts_tablet_manager.cc:902] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea: Initiating tablet copy from peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923)
I20240503 02:15:04.622655 8949 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.628432 11753 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.631443 9082 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.632050 11753 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.634718 11753 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.636000 8820 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:04.637861 8949 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.639430 8267 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fe35d536ab6348c58e90d680383bb763 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:04.640385 9082 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.645483 10591 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:04.645979 8275 catalog_manager.cc:5484] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New cstate: current_term: 1 leader_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.648892 8425 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.650000 8556 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.650647 9212 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:04.652941 8820 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
W20240503 02:15:04.664930 8624 consensus_peers.cc:485] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 -> Peer dabeea74472746538132a520710248ea (127.23.104.202:37759): Couldn't send request to peer dabeea74472746538132a520710248ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe35d536ab6348c58e90d680383bb763. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:04.668108 8972 tablet_copy_service.cc:140] P 99be4ff1114d4aaabcc537604fce54b3: Received BeginTabletCopySession request for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 from peer e512cb0d8f244ac6ab0e7150a0a8e2df ({username='slave'} at 127.23.104.200:41679)
I20240503 02:15:04.668357 8972 tablet_copy_service.cc:161] P 99be4ff1114d4aaabcc537604fce54b3: Beginning new tablet copy session on tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 from peer e512cb0d8f244ac6ab0e7150a0a8e2df at {username='slave'} at 127.23.104.200:41679: session id = e512cb0d8f244ac6ab0e7150a0a8e2df-3c28f0e30cfe46fda64dd2b0fd7847c6
I20240503 02:15:04.671847 8972 tablet_copy_source_session.cc:205] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:04.674163 11749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:04.681372 11749 tablet_copy_client.cc:783] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:04.681733 11749 tablet_copy_client.cc:647] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:04.683727 11749 tablet_copy_client.cc:523] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:04.687600 11749 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap starting.
I20240503 02:15:04.738955 11737 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:04.739457 11737 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap complete.
I20240503 02:15:04.745555 11718 tablet_copy_client.cc:308] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:45923
I20240503 02:15:04.746297 8578 tablet_copy_service.cc:140] P 2a1893ab239f4163a06bdbf8ce579431: Received BeginTabletCopySession request for tablet 7d1f333099d942ae897224a1ee1538a6 from peer dabeea74472746538132a520710248ea ({username='slave'} at 127.23.104.202:33615)
I20240503 02:15:04.746477 11737 ts_tablet_manager.cc:1367] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent bootstrapping tablet: real 0.265s user 0.061s sys 0.044s
I20240503 02:15:04.746490 8578 tablet_copy_service.cc:161] P 2a1893ab239f4163a06bdbf8ce579431: Beginning new tablet copy session on tablet 7d1f333099d942ae897224a1ee1538a6 from peer dabeea74472746538132a520710248ea at {username='slave'} at 127.23.104.202:33615: session id = dabeea74472746538132a520710248ea-7d1f333099d942ae897224a1ee1538a6
I20240503 02:15:04.747282 11737 raft_consensus.cc:348] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:04.747618 11737 raft_consensus.cc:729] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e512cb0d8f244ac6ab0e7150a0a8e2df, State: Initialized, Role: LEARNER
I20240503 02:15:04.749567 8578 tablet_copy_source_session.cc:205] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:04.750962 11718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1f333099d942ae897224a1ee1538a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:04.757885 11718 tablet_copy_client.cc:783] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:04.758117 11718 tablet_copy_client.cc:647] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:04.759655 11718 tablet_copy_client.cc:523] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:04.763397 11737 consensus_queue.cc:260] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:04.764355 11718 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea: Bootstrap starting.
I20240503 02:15:04.765779 11737 ts_tablet_manager.cc:1398] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240503 02:15:04.766448 9234 tablet_copy_service.cc:342] P cb90f34dc37b4b16b2a5b5a4ee012865: Request end of tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-ac90f5b4e0fc43e99b3f76fe92610153 received from {username='slave'} at 127.23.104.200:54679
I20240503 02:15:04.766588 9234 tablet_copy_service.cc:433] P cb90f34dc37b4b16b2a5b5a4ee012865: ending tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-ac90f5b4e0fc43e99b3f76fe92610153 on tablet ac90f5b4e0fc43e99b3f76fe92610153 with peer e512cb0d8f244ac6ab0e7150a0a8e2df
I20240503 02:15:04.789525 11749 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:04.790069 11749 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap complete.
I20240503 02:15:04.790346 11749 ts_tablet_manager.cc:1367] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent bootstrapping tablet: real 0.103s user 0.052s sys 0.004s
I20240503 02:15:04.790977 11749 raft_consensus.cc:348] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:04.791268 11749 raft_consensus.cc:729] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e512cb0d8f244ac6ab0e7150a0a8e2df, State: Initialized, Role: LEARNER
I20240503 02:15:04.791487 11749 consensus_queue.cc:260] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:04.792146 11749 ts_tablet_manager.cc:1398] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:15:04.792922 8972 tablet_copy_service.cc:342] P 99be4ff1114d4aaabcc537604fce54b3: Request end of tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-3c28f0e30cfe46fda64dd2b0fd7847c6 received from {username='slave'} at 127.23.104.200:41679
I20240503 02:15:04.793079 8972 tablet_copy_service.cc:433] P 99be4ff1114d4aaabcc537604fce54b3: ending tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-3c28f0e30cfe46fda64dd2b0fd7847c6 on tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 with peer e512cb0d8f244ac6ab0e7150a0a8e2df
I20240503 02:15:04.833254 11718 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:04.833727 11718 tablet_bootstrap.cc:492] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea: Bootstrap complete.
I20240503 02:15:04.833984 11718 ts_tablet_manager.cc:1367] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea: Time spent bootstrapping tablet: real 0.070s user 0.048s sys 0.009s
I20240503 02:15:04.834772 11718 raft_consensus.cc:348] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:04.834992 11718 raft_consensus.cc:729] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dabeea74472746538132a520710248ea, State: Initialized, Role: LEARNER
I20240503 02:15:04.835229 11718 consensus_queue.cc:260] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:04.836135 11718 ts_tablet_manager.cc:1398] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:15:04.838315 8578 tablet_copy_service.cc:342] P 2a1893ab239f4163a06bdbf8ce579431: Request end of tablet copy session dabeea74472746538132a520710248ea-7d1f333099d942ae897224a1ee1538a6 received from {username='slave'} at 127.23.104.202:33615
I20240503 02:15:04.838441 8578 tablet_copy_service.cc:433] P 2a1893ab239f4163a06bdbf8ce579431: ending tablet copy session dabeea74472746538132a520710248ea-7d1f333099d942ae897224a1ee1538a6 on tablet 7d1f333099d942ae897224a1ee1538a6 with peer dabeea74472746538132a520710248ea
I20240503 02:15:04.869081 10816 raft_consensus.cc:1203] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:04.882072 11079 raft_consensus.cc:1203] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:04.950277 10816 raft_consensus.cc:1203] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:04.969138 11718 ts_tablet_manager.cc:902] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea: Initiating tablet copy from peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403)
I20240503 02:15:04.970259 11718 tablet_copy_client.cc:308] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.197:45403
I20240503 02:15:04.976439 8972 tablet_copy_service.cc:140] P 99be4ff1114d4aaabcc537604fce54b3: Received BeginTabletCopySession request for tablet 64c52f9977ae4eb4ac803be99b6426c2 from peer dabeea74472746538132a520710248ea ({username='slave'} at 127.23.104.202:55513)
I20240503 02:15:04.976619 8972 tablet_copy_service.cc:161] P 99be4ff1114d4aaabcc537604fce54b3: Beginning new tablet copy session on tablet 64c52f9977ae4eb4ac803be99b6426c2 from peer dabeea74472746538132a520710248ea at {username='slave'} at 127.23.104.202:55513: session id = dabeea74472746538132a520710248ea-64c52f9977ae4eb4ac803be99b6426c2
I20240503 02:15:04.978736 8972 tablet_copy_source_session.cc:205] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:04.979948 11718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c52f9977ae4eb4ac803be99b6426c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:04.984633 11718 tablet_copy_client.cc:783] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:04.984989 11718 tablet_copy_client.cc:647] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:04.986693 11718 tablet_copy_client.cc:523] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:04.989370 11718 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea: Bootstrap starting.
I20240503 02:15:05.022346 11718 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:05.022717 11718 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea: Bootstrap complete.
I20240503 02:15:05.022953 11718 ts_tablet_manager.cc:1367] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.001s
I20240503 02:15:05.023692 11718 raft_consensus.cc:348] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:05.024045 11718 raft_consensus.cc:729] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dabeea74472746538132a520710248ea, State: Initialized, Role: LEARNER
I20240503 02:15:05.024276 11718 consensus_queue.cc:260] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:05.025012 11718 ts_tablet_manager.cc:1398] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 02:15:05.025823 8972 tablet_copy_service.cc:342] P 99be4ff1114d4aaabcc537604fce54b3: Request end of tablet copy session dabeea74472746538132a520710248ea-64c52f9977ae4eb4ac803be99b6426c2 received from {username='slave'} at 127.23.104.202:55513
I20240503 02:15:05.025934 8972 tablet_copy_service.cc:433] P 99be4ff1114d4aaabcc537604fce54b3: ending tablet copy session dabeea74472746538132a520710248ea-64c52f9977ae4eb4ac803be99b6426c2 on tablet 64c52f9977ae4eb4ac803be99b6426c2 with peer dabeea74472746538132a520710248ea
I20240503 02:15:05.107172 11079 raft_consensus.cc:1203] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:05.136303 11718 ts_tablet_manager.cc:902] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea: Initiating tablet copy from peer 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:15:05.139048 11718 tablet_copy_client.cc:308] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:45495
I20240503 02:15:05.145025 8710 tablet_copy_service.cc:140] P 846e89dd0b154d2b9bb7d4f315c9fc12: Received BeginTabletCopySession request for tablet fe35d536ab6348c58e90d680383bb763 from peer dabeea74472746538132a520710248ea ({username='slave'} at 127.23.104.202:37369)
I20240503 02:15:05.145242 8710 tablet_copy_service.cc:161] P 846e89dd0b154d2b9bb7d4f315c9fc12: Beginning new tablet copy session on tablet fe35d536ab6348c58e90d680383bb763 from peer dabeea74472746538132a520710248ea at {username='slave'} at 127.23.104.202:37369: session id = dabeea74472746538132a520710248ea-fe35d536ab6348c58e90d680383bb763
I20240503 02:15:05.147991 8710 tablet_copy_source_session.cc:205] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:05.149757 11718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe35d536ab6348c58e90d680383bb763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:05.154485 11718 tablet_copy_client.cc:783] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:05.154688 11718 tablet_copy_client.cc:647] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:05.157017 11718 tablet_copy_client.cc:523] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:05.159346 11718 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea: Bootstrap starting.
I20240503 02:15:05.191274 11718 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:05.191673 11718 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea: Bootstrap complete.
I20240503 02:15:05.191948 11718 ts_tablet_manager.cc:1367] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea: Time spent bootstrapping tablet: real 0.033s user 0.031s sys 0.004s
I20240503 02:15:05.192744 11718 raft_consensus.cc:348] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:05.193069 11718 raft_consensus.cc:729] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dabeea74472746538132a520710248ea, State: Initialized, Role: LEARNER
I20240503 02:15:05.193303 11718 consensus_queue.cc:260] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:05.193809 11718 ts_tablet_manager.cc:1398] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:15:05.194598 8710 tablet_copy_service.cc:342] P 846e89dd0b154d2b9bb7d4f315c9fc12: Request end of tablet copy session dabeea74472746538132a520710248ea-fe35d536ab6348c58e90d680383bb763 received from {username='slave'} at 127.23.104.202:37369
I20240503 02:15:05.194752 8710 tablet_copy_service.cc:433] P 846e89dd0b154d2b9bb7d4f315c9fc12: ending tablet copy session dabeea74472746538132a520710248ea-fe35d536ab6348c58e90d680383bb763 on tablet fe35d536ab6348c58e90d680383bb763 with peer dabeea74472746538132a520710248ea
I20240503 02:15:05.330969 11769 raft_consensus.cc:1050] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865: attempting to promote NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df to VOTER
I20240503 02:15:05.332072 11769 consensus_queue.cc:237] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:05.335465 10816 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.336084 8687 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.336264 8424 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.336441 8556 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.337222 11662 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.337520 11769 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.337993 11770 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.338212 8819 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.338307 11769 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.339839 9082 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.340025 8949 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.340644 11877 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.340970 11770 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.341346 11662 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.377178 11877 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.378046 8425 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.383445 10816 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.384502 8819 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.386598 8276 catalog_manager.cc:5484] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.389864 8949 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.392668 9083 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.398491 8556 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.400367 11079 raft_consensus.cc:1203] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:05.402233 8689 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.406548 11630 raft_consensus.cc:1050] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431: attempting to promote NON_VOTER dabeea74472746538132a520710248ea to VOTER
I20240503 02:15:05.407827 11630 consensus_queue.cc:237] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:05.429761 8424 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.429821 9213 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.430136 8819 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.430366 11079 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea [term 1 LEARNER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.431066 8689 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.430981 11628 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.431396 11561 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.432066 11628 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.433763 11561 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.434352 11561 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.440196 9082 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.440907 11561 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.442181 9212 consensus_queue.cc:237] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:05.449054 11224 raft_consensus.cc:1050] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3: attempting to promote NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df to VOTER
I20240503 02:15:05.450660 11224 consensus_queue.cc:237] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:05.455178 10816 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.455909 8690 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.455837 11769 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.456696 8556 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.457036 11769 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.457464 11877 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.458690 8425 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.458712 9083 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.459314 11769 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.459928 8820 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.460695 11877 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.465159 11662 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.465845 11770 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.467535 8556 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.468329 8425 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.468525 8819 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.475551 8279 catalog_manager.cc:5484] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf reported cstate change: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.476411 8269 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac90f5b4e0fc43e99b3f76fe92610153 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:05.477576 10816 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.481019 8690 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.481626 8931 tablet_service.cc:1514] Processing DeleteTablet for tablet ac90f5b4e0fc43e99b3f76fe92610153 with delete_type TABLET_DATA_TOMBSTONED (TS 99be4ff1114d4aaabcc537604fce54b3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52056
W20240503 02:15:05.483598 8492 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.202:37759, user_credentials={real_user=slave}} blocked reactor thread for 52744.8us
I20240503 02:15:05.484489 8949 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.485399 11630 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.487727 9083 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.490291 11902 tablet_replica.cc:330] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3: stopping tablet replica
I20240503 02:15:05.490525 11902 raft_consensus.cc:2229] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:05.490993 11902 raft_consensus.cc:2258] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:05.492256 11902 ts_tablet_manager.cc:1875] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:05.500288 11902 ts_tablet_manager.cc:1888] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:05.500572 11902 log.cc:1196] T ac90f5b4e0fc43e99b3f76fe92610153 P 99be4ff1114d4aaabcc537604fce54b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/wals/ac90f5b4e0fc43e99b3f76fe92610153
I20240503 02:15:05.501610 8268 catalog_manager.cc:4830] TS 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): tablet ac90f5b4e0fc43e99b3f76fe92610153 (table rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]) successfully deleted
I20240503 02:15:05.514422 11078 raft_consensus.cc:1203] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:05.533396 11079 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.532878 11894 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.536926 9212 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.539374 8689 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.540882 8424 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
W20240503 02:15:05.542238 8756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:45403, user_credentials={real_user=slave}} blocked reactor thread for 53006us
I20240503 02:15:05.542512 8949 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.544714 8273 catalog_manager.cc:5484] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.548053 9083 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.548578 8819 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.548609 8820 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.549670 11676 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.552181 11668 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.553551 9083 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.556849 10816 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.557251 8690 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.557829 11676 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.558377 11676 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.570416 8556 consensus_queue.cc:237] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:05.571087 9212 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.571774 11668 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.573967 11079 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.574559 11628 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.574926 8425 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.575862 8689 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.576853 11628 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.583910 9214 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.584282 8819 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.584975 11628 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.586064 11628 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.589457 11628 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.592574 11894 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.592983 9214 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.596874 8269 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 493e2b13e19241fa8fa48c367986caaf with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:05.597707 8690 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.598233 8424 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.600286 11079 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.604804 9081 raft_consensus.cc:1261] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.605794 8282 catalog_manager.cc:5484] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:15:05.608232 11630 consensus_queue.cc:1035] T 493e2b13e19241fa8fa48c367986caaf P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.612361 8931 tablet_service.cc:1514] Processing DeleteTablet for tablet 493e2b13e19241fa8fa48c367986caaf with delete_type TABLET_DATA_TOMBSTONED (TS 99be4ff1114d4aaabcc537604fce54b3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52056
I20240503 02:15:05.612845 11902 tablet_replica.cc:330] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3: stopping tablet replica
I20240503 02:15:05.613130 11902 raft_consensus.cc:2229] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:05.612514 9083 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.613570 11902 raft_consensus.cc:2258] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:05.614950 11902 ts_tablet_manager.cc:1875] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:05.619896 11676 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.625460 11902 ts_tablet_manager.cc:1888] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:05.625682 11902 log.cc:1196] T 493e2b13e19241fa8fa48c367986caaf P 99be4ff1114d4aaabcc537604fce54b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/wals/493e2b13e19241fa8fa48c367986caaf
I20240503 02:15:05.626576 8268 catalog_manager.cc:4830] TS 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): tablet 493e2b13e19241fa8fa48c367986caaf (table rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]) successfully deleted
I20240503 02:15:05.627099 8556 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.627826 11676 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.628180 9213 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.628271 10816 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.636409 8556 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.637679 8279 catalog_manager.cc:5484] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.639305 8820 raft_consensus.cc:2932] T 493e2b13e19241fa8fa48c367986caaf P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.645421 8424 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.645962 11911 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.645778 8690 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.647208 8820 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.648825 9082 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.652261 8424 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:05.677361 11561 raft_consensus.cc:1050] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431: attempting to promote NON_VOTER dabeea74472746538132a520710248ea to VOTER
I20240503 02:15:05.678644 11561 consensus_queue.cc:237] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:05.681983 9213 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.682302 9082 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.682466 11079 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.682540 11630 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.682848 11628 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.683900 11628 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.685824 8690 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.686091 8424 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.688500 11917 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.688978 11917 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.698248 8949 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.705319 8820 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.707362 11561 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.708559 11628 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.736042 11628 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.738468 8425 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.741604 8819 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.744547 8949 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.745030 8274 catalog_manager.cc:5484] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.748369 9213 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.751664 9083 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.751670 8690 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.760964 11079 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.765250 8556 consensus_queue.cc:237] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:05.768080 9082 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.768410 9213 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.768599 11630 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.769011 11917 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.770318 11078 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.771451 8819 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.771776 8424 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.771996 8690 raft_consensus.cc:1261] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.772765 11922 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240503 02:15:05.773087 11628 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.773396 11561 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.773860 11917 consensus_queue.cc:1035] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.777241 11917 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.780455 8819 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.780486 8424 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.781069 11676 raft_consensus.cc:1050] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3: attempting to promote NON_VOTER dabeea74472746538132a520710248ea to VOTER
I20240503 02:15:05.782528 11676 consensus_queue.cc:237] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:05.785341 8269 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d1f333099d942ae897224a1ee1538a6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:05.785995 9214 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.787032 9083 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.787171 8274 catalog_manager.cc:5484] T 7d1f333099d942ae897224a1ee1538a6 P 79e41f25066e4e5abbd71e48d545c6bf reported cstate change: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.790697 8690 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.793844 8931 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1f333099d942ae897224a1ee1538a6 with delete_type TABLET_DATA_TOMBSTONED (TS 99be4ff1114d4aaabcc537604fce54b3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52056
I20240503 02:15:05.794248 11902 tablet_replica.cc:330] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3: stopping tablet replica
I20240503 02:15:05.794440 11902 raft_consensus.cc:2229] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:05.794706 11902 raft_consensus.cc:2258] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:05.798574 11902 ts_tablet_manager.cc:1875] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:05.799111 11079 raft_consensus.cc:2932] T 7d1f333099d942ae897224a1ee1538a6 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.800189 8689 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.800988 8819 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.802125 11078 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.802176 9214 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.802572 11918 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.802904 11224 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.803354 11668 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.804533 11918 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.807641 8424 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.808349 11676 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.812891 8556 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.813530 11676 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.827212 9083 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.828660 11676 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.840857 11902 ts_tablet_manager.cc:1888] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:05.841187 11902 log.cc:1196] T 7d1f333099d942ae897224a1ee1538a6 P 99be4ff1114d4aaabcc537604fce54b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/wals/7d1f333099d942ae897224a1ee1538a6
I20240503 02:15:05.844383 8268 catalog_manager.cc:4830] TS 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): tablet 7d1f333099d942ae897224a1ee1538a6 (table rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]) successfully deleted
I20240503 02:15:05.867048 8556 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.867419 11676 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.872519 8690 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.880048 9082 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.884634 8279 catalog_manager.cc:5484] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:15:05.884928 9214 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.889935 8949 consensus_queue.cc:237] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:05.895018 8424 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.899953 9214 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.901432 9082 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.904595 8690 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.906461 11911 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.907886 11918 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.908422 11918 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.881238 11078 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.924914 8690 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.925858 11942 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.931174 11079 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.932854 11676 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.934844 11079 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.937131 8557 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.937513 8424 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:05.937682 11676 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.937987 11911 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:05.939190 8557 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.941427 8268 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 64c52f9977ae4eb4ac803be99b6426c2 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:05.941543 9214 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.943075 8277 catalog_manager.cc:5484] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:05.943660 9082 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.944638 8424 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.950417 8801 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c52f9977ae4eb4ac803be99b6426c2 with delete_type TABLET_DATA_TOMBSTONED (TS 79e41f25066e4e5abbd71e48d545c6bf not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38550
I20240503 02:15:05.951810 11946 tablet_replica.cc:330] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: stopping tablet replica
I20240503 02:15:05.952112 11946 raft_consensus.cc:2229] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:05.952533 11946 pending_rounds.cc:62] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: Trying to abort 1 pending ops.
I20240503 02:15:05.952685 11946 pending_rounds.cc:68] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023421463680708608 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "64c52f9977ae4eb4ac803be99b6426c2" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } } }
I20240503 02:15:05.953533 11946 raft_consensus.cc:2894] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240503 02:15:05.953708 11946 raft_consensus.cc:2866] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:15:05.953857 11946 raft_consensus.cc:2258] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:05.956178 11946 ts_tablet_manager.cc:1875] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:05.962092 11773 raft_consensus.cc:1050] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12: attempting to promote NON_VOTER dabeea74472746538132a520710248ea to VOTER
I20240503 02:15:05.963254 11773 consensus_queue.cc:237] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:05.964440 11946 ts_tablet_manager.cc:1888] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:05.964629 11946 log.cc:1196] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/64c52f9977ae4eb4ac803be99b6426c2
I20240503 02:15:05.965637 8267 catalog_manager.cc:4830] TS 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747): tablet 64c52f9977ae4eb4ac803be99b6426c2 (table rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]) successfully deleted
I20240503 02:15:05.972944 11079 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.973459 11855 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.973590 9214 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.974300 11854 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.974623 8949 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.975176 11854 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.980155 8819 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.980719 11854 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.993263 9082 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.993722 11854 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.995743 8557 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.996245 11854 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:05.997664 11854 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.998865 8424 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:05.999325 8950 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:05.999992 11855 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:06.009254 8275 catalog_manager.cc:5484] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:06.014086 9081 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:06.017017 8557 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:06.023546 11079 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:06.023965 8424 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:06.027460 9214 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:06.028432 8819 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:06.254410 8950 consensus_queue.cc:237] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:06.257009 9081 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:06.257231 10816 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:06.257759 8557 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:06.257822 11224 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:06.258311 11918 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:06.258823 8690 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:06.259842 9214 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:06.260267 11918 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:06.260653 11676 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:06.264695 11224 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:06.264755 8424 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:06.263835 11668 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:06.265239 9213 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:06.266021 9081 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:06.269027 10816 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:06.270226 8268 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:06.273811 11676 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:06.273481 8557 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:06.275231 8690 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:06.275882 8275 catalog_manager.cc:5484] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d reported cstate change: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:06.275761 8424 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:06.283285 8801 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 with delete_type TABLET_DATA_TOMBSTONED (TS 79e41f25066e4e5abbd71e48d545c6bf not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38550
I20240503 02:15:06.284067 11946 tablet_replica.cc:330] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf: stopping tablet replica
I20240503 02:15:06.284303 11946 raft_consensus.cc:2229] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:06.284579 11946 raft_consensus.cc:2258] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:06.285758 11946 ts_tablet_manager.cc:1875] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:06.301210 11946 ts_tablet_manager.cc:1888] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:06.301445 11946 log.cc:1196] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/3c28f0e30cfe46fda64dd2b0fd7847c6
I20240503 02:15:06.302224 8267 catalog_manager.cc:4830] TS 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747): tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 (table rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]) successfully deleted
I20240503 02:15:06.314183 8557 consensus_queue.cc:237] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:15:06.316931 8690 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.317237 8424 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.317636 9213 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.318145 11628 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.318217 9081 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.318408 11630 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.318470 8819 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.318739 11917 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.319067 8950 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.320066 11917 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.320556 11561 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.321619 11917 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.343919 11917 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.347537 8425 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.350339 8819 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.352288 8275 catalog_manager.cc:5484] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:06.355677 8950 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.360363 8557 consensus_queue.cc:237] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:06.360456 9213 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.363906 9082 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.364033 8424 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.364878 8950 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.364789 11628 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.365043 8819 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.365648 9081 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.367182 11561 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.368150 11628 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.367558 8690 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.369796 11628 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.370354 9214 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.373050 11628 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.376571 11628 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:06.383492 8689 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.383323 8950 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:06.388196 8424 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:06.388177 8819 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:06.388628 8275 catalog_manager.cc:5484] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:06.392582 11561 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.394469 8690 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:06.405678 8269 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:06.407179 9214 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
W20240503 02:15:06.413813 8493 consensus_peers.cc:485] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 -> Peer e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200:45027): Couldn't send request to peer e512cb0d8f244ac6ab0e7150a0a8e2df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee0464bacc4a4bd39dde43ccf4d9a0fc. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:06.415488 9081 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:06.420922 9081 consensus_queue.cc:237] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:15:06.423595 8557 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.424098 9214 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.424433 8950 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.424453 8424 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.424643 8819 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.424960 11510 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.425326 11984 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.425720 12008 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.425987 12010 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.427820 8690 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.435979 12009 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.436445 11984 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.437655 12009 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.437736 8556 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.444636 8950 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.445091 8274 catalog_manager.cc:5484] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.445147 9214 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.450228 8690 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.456574 8424 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.465103 8687 consensus_queue.cc:237] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:06.466720 9081 consensus_queue.cc:237] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:06.466284 8819 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.468451 11079 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:06.469069 11953 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:06.469695 8950 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.469919 8556 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.470378 12008 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.470839 12008 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.471107 9214 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.472690 9214 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:06.473181 11953 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:06.473240 8949 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:06.474057 11854 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:06.474551 8819 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.475244 9081 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:06.476316 8425 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:06.476857 11773 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:06.477337 11773 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:06.478329 12008 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.480733 11855 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:06.482496 8425 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:06.485692 8557 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:06.485849 9083 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:06.491365 11984 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.494149 8690 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:15:06.494439 9017 consensus_peers.cc:485] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d -> Peer 4792082063bd4d58bc222c9c95b16335 (127.23.104.201:46877): Couldn't send request to peer 4792082063bd4d58bc222c9c95b16335. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b71357f9f8ef449189f485b3246869f1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:06.494525 11773 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:06.494891 9213 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:06.497711 8556 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:06.500366 11984 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:06.501219 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe35d536ab6348c58e90d680383bb763 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:06.502365 11079 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:06.504472 8950 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:06.504813 8949 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:06.508761 8282 catalog_manager.cc:5484] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:06.514824 8424 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.515595 8267 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b71357f9f8ef449189f485b3246869f1 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:06.516280 9214 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:06.518980 8557 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:06.521473 8277 catalog_manager.cc:5484] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:06.522017 11984 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.522543 12009 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.525310 8801 tablet_service.cc:1514] Processing DeleteTablet for tablet fe35d536ab6348c58e90d680383bb763 with delete_type TABLET_DATA_TOMBSTONED (TS 79e41f25066e4e5abbd71e48d545c6bf not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38550
I20240503 02:15:06.525672 11946 tablet_replica.cc:330] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf: stopping tablet replica
I20240503 02:15:06.525888 11946 raft_consensus.cc:2229] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:06.526237 11946 raft_consensus.cc:2258] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:06.527333 11946 ts_tablet_manager.cc:1875] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:06.527033 8687 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:06.527251 8425 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:06.536830 11946 ts_tablet_manager.cc:1888] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:06.537024 11946 log.cc:1196] T fe35d536ab6348c58e90d680383bb763 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/fe35d536ab6348c58e90d680383bb763
I20240503 02:15:06.537781 8267 catalog_manager.cc:4830] TS 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747): tablet fe35d536ab6348c58e90d680383bb763 (table rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]) successfully deleted
I20240503 02:15:06.542266 8819 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:06.557040 8425 consensus_queue.cc:237] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:15:06.560642 8557 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.560691 8687 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.561443 11358 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.561837 12027 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.563335 9213 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.564872 11358 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.571075 9083 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.572221 12027 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.572247 8950 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.572955 11358 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.573290 8819 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.573876 11358 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.575858 11358 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.582639 9213 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.586186 8687 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.588222 8557 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.592746 8282 catalog_manager.cc:5484] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.597831 8819 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.606890 8425 consensus_queue.cc:237] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:06.606746 8950 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.610225 9213 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.615089 8819 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.615430 8557 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.615679 12035 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.616644 8950 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.617178 12036 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.620059 8687 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.620817 12036 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.621414 9083 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.622813 12036 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.623162 8819 consensus_queue.cc:237] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:15:06.624349 9083 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.624693 8687 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:06.628054 12028 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 02:15:06.629024 12028 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.628515 12036 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:06.630813 9213 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:06.630960 8275 catalog_manager.cc:5484] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 reported cstate change: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:06.635491 8557 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:06.639532 8267 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1ea7444883c642419cd3e509f2b8d090 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:06.640451 8950 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:06.640430 8820 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:06.642325 8687 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.642771 8425 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.643386 8557 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.644004 9083 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.644522 11906 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.644690 9213 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.645032 11906 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.645094 9081 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:06.661296 12047 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.661991 12047 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:15:06.669795 8360 consensus_peers.cc:485] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c -> Peer e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200:45027): Couldn't send request to peer e512cb0d8f244ac6ab0e7150a0a8e2df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ea7444883c642419cd3e509f2b8d090. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:06.671190 12049 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.673090 8950 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:06.675973 12047 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.706600 12048 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.707340 8950 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.719012 8557 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.719177 8279 catalog_manager.cc:5484] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "79e41f25066e4e5abbd71e48d545c6bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:06.720815 9213 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.728474 8820 consensus_queue.cc:237] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:06.729988 9081 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.731588 8950 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.732249 8556 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.732545 12047 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.733888 9214 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.734071 12048 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.734511 12048 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:06.738353 12047 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:06.743891 8556 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:06.751056 8279 catalog_manager.cc:5484] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "79e41f25066e4e5abbd71e48d545c6bf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:06.763020 9081 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.763742 8267 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3bca5fd721f84f38bff08308965eb33f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:06.763886 8425 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.764361 12048 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 02:15:06.766605 8687 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:06.767268 12049 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.767063 8950 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:06.767859 9214 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:06.768419 8687 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.772346 9081 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:06.777043 12057 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:06.779145 8425 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:06.782949 8425 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:06.782884 8687 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
W20240503 02:15:06.785710 8755 consensus_peers.cc:485] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf -> Peer 4792082063bd4d58bc222c9c95b16335 (127.23.104.201:46877): Couldn't send request to peer 4792082063bd4d58bc222c9c95b16335. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bca5fd721f84f38bff08308965eb33f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:06.828174 12074 ts_tablet_manager.cc:902] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df: Initiating tablet copy from peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923)
I20240503 02:15:06.844105 12074 tablet_copy_client.cc:308] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:45923
I20240503 02:15:06.858994 8578 tablet_copy_service.cc:140] P 2a1893ab239f4163a06bdbf8ce579431: Received BeginTabletCopySession request for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc from peer e512cb0d8f244ac6ab0e7150a0a8e2df ({username='slave'} at 127.23.104.200:41677)
I20240503 02:15:06.859177 8578 tablet_copy_service.cc:161] P 2a1893ab239f4163a06bdbf8ce579431: Beginning new tablet copy session on tablet ee0464bacc4a4bd39dde43ccf4d9a0fc from peer e512cb0d8f244ac6ab0e7150a0a8e2df at {username='slave'} at 127.23.104.200:41677: session id = e512cb0d8f244ac6ab0e7150a0a8e2df-ee0464bacc4a4bd39dde43ccf4d9a0fc
I20240503 02:15:06.861378 8578 tablet_copy_source_session.cc:205] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:06.862736 12074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:06.868513 12074 tablet_copy_client.cc:783] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:06.868772 12074 tablet_copy_client.cc:647] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:06.870851 12074 tablet_copy_client.cc:523] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:06.874186 12074 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap starting.
I20240503 02:15:06.892026 12100 ts_tablet_manager.cc:902] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335: Initiating tablet copy from peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:15:06.913758 12100 tablet_copy_client.cc:308] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:41237
I20240503 02:15:06.914461 9103 tablet_copy_service.cc:140] P c399efcbba924863ac89d156dc07ac7d: Received BeginTabletCopySession request for tablet b71357f9f8ef449189f485b3246869f1 from peer 4792082063bd4d58bc222c9c95b16335 ({username='slave'} at 127.23.104.201:41501)
I20240503 02:15:06.914642 9103 tablet_copy_service.cc:161] P c399efcbba924863ac89d156dc07ac7d: Beginning new tablet copy session on tablet b71357f9f8ef449189f485b3246869f1 from peer 4792082063bd4d58bc222c9c95b16335 at {username='slave'} at 127.23.104.201:41501: session id = 4792082063bd4d58bc222c9c95b16335-b71357f9f8ef449189f485b3246869f1
I20240503 02:15:06.917501 9103 tablet_copy_source_session.cc:205] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:06.918809 12100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71357f9f8ef449189f485b3246869f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:06.925212 12100 tablet_copy_client.cc:783] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:06.925515 12100 tablet_copy_client.cc:647] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:06.927232 12074 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:06.927239 12100 tablet_copy_client.cc:523] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:06.927598 12074 tablet_bootstrap.cc:492] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap complete.
I20240503 02:15:06.927881 12074 ts_tablet_manager.cc:1367] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent bootstrapping tablet: real 0.054s user 0.046s sys 0.005s
I20240503 02:15:06.928512 12074 raft_consensus.cc:348] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:06.928756 12074 raft_consensus.cc:729] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e512cb0d8f244ac6ab0e7150a0a8e2df, State: Initialized, Role: LEARNER
I20240503 02:15:06.928951 12074 consensus_queue.cc:260] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:06.929811 12074 ts_tablet_manager.cc:1398] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:15:06.930567 8578 tablet_copy_service.cc:342] P 2a1893ab239f4163a06bdbf8ce579431: Request end of tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-ee0464bacc4a4bd39dde43ccf4d9a0fc received from {username='slave'} at 127.23.104.200:41677
I20240503 02:15:06.930699 8578 tablet_copy_service.cc:433] P 2a1893ab239f4163a06bdbf8ce579431: ending tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-ee0464bacc4a4bd39dde43ccf4d9a0fc on tablet ee0464bacc4a4bd39dde43ccf4d9a0fc with peer e512cb0d8f244ac6ab0e7150a0a8e2df
I20240503 02:15:06.931416 12100 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335: Bootstrap starting.
I20240503 02:15:06.977384 12100 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:06.977836 12100 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335: Bootstrap complete.
I20240503 02:15:06.978089 12100 ts_tablet_manager.cc:1367] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335: Time spent bootstrapping tablet: real 0.047s user 0.035s sys 0.008s
I20240503 02:15:06.978732 12100 raft_consensus.cc:348] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:06.978981 12100 raft_consensus.cc:729] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4792082063bd4d58bc222c9c95b16335, State: Initialized, Role: LEARNER
I20240503 02:15:06.979166 12100 consensus_queue.cc:260] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:06.980036 12100 ts_tablet_manager.cc:1398] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240503 02:15:06.980770 9103 tablet_copy_service.cc:342] P c399efcbba924863ac89d156dc07ac7d: Request end of tablet copy session 4792082063bd4d58bc222c9c95b16335-b71357f9f8ef449189f485b3246869f1 received from {username='slave'} at 127.23.104.201:41501
I20240503 02:15:06.980913 9103 tablet_copy_service.cc:433] P c399efcbba924863ac89d156dc07ac7d: ending tablet copy session 4792082063bd4d58bc222c9c95b16335-b71357f9f8ef449189f485b3246869f1 on tablet b71357f9f8ef449189f485b3246869f1 with peer 4792082063bd4d58bc222c9c95b16335
I20240503 02:15:07.107162 12074 ts_tablet_manager.cc:902] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df: Initiating tablet copy from peer 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635)
I20240503 02:15:07.111480 8950 consensus_queue.cc:237] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:15:07.121528 9081 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:07.121623 8820 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:07.122278 8425 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:07.122531 12074 tablet_copy_client.cc:308] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.193:38635
I20240503 02:15:07.123041 11668 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:07.124198 11668 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:07.124354 9214 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:07.125008 8557 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:07.125103 11918 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:07.125749 11918 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:07.127770 8687 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:07.128099 11918 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:07.128424 11676 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:07.135339 8445 tablet_copy_service.cc:140] P 4c668b2e2d5c4edba46cbc5c36065c0c: Received BeginTabletCopySession request for tablet 1ea7444883c642419cd3e509f2b8d090 from peer e512cb0d8f244ac6ab0e7150a0a8e2df ({username='slave'} at 127.23.104.200:53811)
I20240503 02:15:07.135633 8445 tablet_copy_service.cc:161] P 4c668b2e2d5c4edba46cbc5c36065c0c: Beginning new tablet copy session on tablet 1ea7444883c642419cd3e509f2b8d090 from peer e512cb0d8f244ac6ab0e7150a0a8e2df at {username='slave'} at 127.23.104.200:53811: session id = e512cb0d8f244ac6ab0e7150a0a8e2df-1ea7444883c642419cd3e509f2b8d090
I20240503 02:15:07.140033 8445 tablet_copy_source_session.cc:205] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:07.144008 11668 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:07.147209 12074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea7444883c642419cd3e509f2b8d090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:07.149819 8275 catalog_manager.cc:5484] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } }
I20240503 02:15:07.149860 9214 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:07.150275 9081 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:07.158020 8687 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:07.163699 12074 tablet_copy_client.cc:783] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:07.163944 12074 tablet_copy_client.cc:647] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:07.165270 8556 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:07.165634 8950 consensus_queue.cc:237] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:07.169481 9214 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:07.170153 11676 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:07.171197 8558 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:07.171996 11676 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:07.176373 12074 tablet_copy_client.cc:523] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:07.180413 12074 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap starting.
I20240503 02:15:07.188815 8687 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:07.190275 8820 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:07.191169 9081 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:07.191488 11676 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:07.191953 11676 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:07.194839 8820 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:07.196655 11676 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:07.197448 8425 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:07.197453 11676 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.199762 12100 ts_tablet_manager.cc:902] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335: Initiating tablet copy from peer 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747)
I20240503 02:15:07.205700 8950 consensus_queue.cc:237] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:15:07.207111 8268 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 95a8f2627c5c49a39291c96eb00fec18 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:07.208400 8558 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.209436 9214 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.214972 8279 catalog_manager.cc:5484] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.217718 8687 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.218946 12100 tablet_copy_client.cc:308] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:34747
I20240503 02:15:07.233891 11668 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:07.236269 9214 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:15:07.236757 8886 consensus_peers.cc:485] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 -> Peer 4792082063bd4d58bc222c9c95b16335 (127.23.104.201:46877): Couldn't send request to peer 4792082063bd4d58bc222c9c95b16335. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95a8f2627c5c49a39291c96eb00fec18. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:07.236856 12137 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:07.237679 8820 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.244537 8841 tablet_copy_service.cc:140] P 79e41f25066e4e5abbd71e48d545c6bf: Received BeginTabletCopySession request for tablet 3bca5fd721f84f38bff08308965eb33f from peer 4792082063bd4d58bc222c9c95b16335 ({username='slave'} at 127.23.104.201:57903)
I20240503 02:15:07.244704 8841 tablet_copy_service.cc:161] P 79e41f25066e4e5abbd71e48d545c6bf: Beginning new tablet copy session on tablet 3bca5fd721f84f38bff08308965eb33f from peer 4792082063bd4d58bc222c9c95b16335 at {username='slave'} at 127.23.104.201:57903: session id = 4792082063bd4d58bc222c9c95b16335-3bca5fd721f84f38bff08308965eb33f
I20240503 02:15:07.247130 9081 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:07.247524 8841 tablet_copy_source_session.cc:205] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:07.248823 12100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bca5fd721f84f38bff08308965eb33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:07.257861 8820 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:07.258874 8687 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:07.259454 8425 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:07.264005 8425 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.268431 8424 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:07.268440 11668 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:07.269170 11918 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:07.271369 11668 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:07.272601 8558 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:07.274782 11668 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:07.281184 9081 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.285478 12137 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:07.282774 12100 tablet_copy_client.cc:783] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:07.289994 12100 tablet_copy_client.cc:647] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:07.292361 12100 tablet_copy_client.cc:523] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:07.292955 10946 raft_consensus.cc:1203] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:07.296247 11668 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:07.299551 8558 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:07.306628 12074 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:07.307075 12074 tablet_bootstrap.cc:492] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap complete.
I20240503 02:15:07.307380 12074 ts_tablet_manager.cc:1367] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent bootstrapping tablet: real 0.127s user 0.050s sys 0.001s
I20240503 02:15:07.307575 9214 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:07.308274 12074 raft_consensus.cc:348] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:07.308579 12074 raft_consensus.cc:729] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e512cb0d8f244ac6ab0e7150a0a8e2df, State: Initialized, Role: LEARNER
I20240503 02:15:07.308823 12074 consensus_queue.cc:260] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:07.310168 12074 ts_tablet_manager.cc:1398] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 02:15:07.311349 8445 tablet_copy_service.cc:342] P 4c668b2e2d5c4edba46cbc5c36065c0c: Request end of tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-1ea7444883c642419cd3e509f2b8d090 received from {username='slave'} at 127.23.104.200:53811
I20240503 02:15:07.311511 8445 tablet_copy_service.cc:433] P 4c668b2e2d5c4edba46cbc5c36065c0c: ending tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-1ea7444883c642419cd3e509f2b8d090 on tablet 1ea7444883c642419cd3e509f2b8d090 with peer e512cb0d8f244ac6ab0e7150a0a8e2df
I20240503 02:15:07.315213 12100 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335: Bootstrap starting.
I20240503 02:15:07.318607 8282 catalog_manager.cc:5484] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:07.320576 8687 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:07.331881 8820 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:07.336169 8425 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
W20240503 02:15:07.343206 11676 consensus_queue.cc:1159] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c399efcbba924863ac89d156dc07ac7d" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 3 last_received_current_leader { term: 1 index: 4 } }
I20240503 02:15:07.344398 8950 consensus_queue.cc:237] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:07.349357 8558 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:07.350356 8424 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:07.350610 11676 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:07.351240 12118 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:15:07.351493 8886 consensus_peers.cc:485] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 -> Peer 4792082063bd4d58bc222c9c95b16335 (127.23.104.201:46877): Couldn't send request to peer 4792082063bd4d58bc222c9c95b16335. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7a434e2df0a4ea0b2ab5c20034033c6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:07.351926 8819 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:07.356328 10816 raft_consensus.cc:1203] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:07.356578 12137 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:07.350396 8687 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:07.361389 9083 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:07.361845 9214 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:07.362454 11676 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:07.362718 12118 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:07.363324 11918 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.365195 9214 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.368880 11676 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20240503 02:15:07.369884 8268 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a7a434e2df0a4ea0b2ab5c20034033c6 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:07.374063 8282 catalog_manager.cc:5484] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.374068 8558 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.385427 9083 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:07.386166 8687 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.392915 8820 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.393255 8425 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.401942 9083 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:07.430764 12100 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:07.431216 12100 tablet_bootstrap.cc:492] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335: Bootstrap complete.
I20240503 02:15:07.431515 12100 ts_tablet_manager.cc:1367] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335: Time spent bootstrapping tablet: real 0.132s user 0.045s sys 0.019s
I20240503 02:15:07.432229 12100 raft_consensus.cc:348] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:07.432469 12100 raft_consensus.cc:729] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4792082063bd4d58bc222c9c95b16335, State: Initialized, Role: LEARNER
I20240503 02:15:07.433185 12100 consensus_queue.cc:260] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:07.434188 12100 ts_tablet_manager.cc:1398] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:15:07.434855 8841 tablet_copy_service.cc:342] P 79e41f25066e4e5abbd71e48d545c6bf: Request end of tablet copy session 4792082063bd4d58bc222c9c95b16335-3bca5fd721f84f38bff08308965eb33f received from {username='slave'} at 127.23.104.201:57903
I20240503 02:15:07.434985 8841 tablet_copy_service.cc:433] P 79e41f25066e4e5abbd71e48d545c6bf: ending tablet copy session 4792082063bd4d58bc222c9c95b16335-3bca5fd721f84f38bff08308965eb33f on tablet 3bca5fd721f84f38bff08308965eb33f with peer 4792082063bd4d58bc222c9c95b16335
I20240503 02:15:07.714516 10946 raft_consensus.cc:1203] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:07.715175 10816 raft_consensus.cc:1203] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:07.785060 12100 ts_tablet_manager.cc:902] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335: Initiating tablet copy from peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403)
I20240503 02:15:07.786443 12100 tablet_copy_client.cc:308] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.197:45403
I20240503 02:15:07.793433 8972 tablet_copy_service.cc:140] P 99be4ff1114d4aaabcc537604fce54b3: Received BeginTabletCopySession request for tablet 95a8f2627c5c49a39291c96eb00fec18 from peer 4792082063bd4d58bc222c9c95b16335 ({username='slave'} at 127.23.104.201:49997)
I20240503 02:15:07.793579 8972 tablet_copy_service.cc:161] P 99be4ff1114d4aaabcc537604fce54b3: Beginning new tablet copy session on tablet 95a8f2627c5c49a39291c96eb00fec18 from peer 4792082063bd4d58bc222c9c95b16335 at {username='slave'} at 127.23.104.201:49997: session id = 4792082063bd4d58bc222c9c95b16335-95a8f2627c5c49a39291c96eb00fec18
I20240503 02:15:07.795524 8972 tablet_copy_source_session.cc:205] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:07.796710 12100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a8f2627c5c49a39291c96eb00fec18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:07.801396 12100 tablet_copy_client.cc:783] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:07.801632 12100 tablet_copy_client.cc:647] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:07.803154 12100 tablet_copy_client.cc:523] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:07.805593 12100 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335: Bootstrap starting.
I20240503 02:15:07.806587 12218 ts_tablet_manager.cc:902] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335: Initiating tablet copy from peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403)
I20240503 02:15:07.808636 12218 tablet_copy_client.cc:308] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.197:45403
I20240503 02:15:07.809314 8972 tablet_copy_service.cc:140] P 99be4ff1114d4aaabcc537604fce54b3: Received BeginTabletCopySession request for tablet a7a434e2df0a4ea0b2ab5c20034033c6 from peer 4792082063bd4d58bc222c9c95b16335 ({username='slave'} at 127.23.104.201:49997)
I20240503 02:15:07.809492 8972 tablet_copy_service.cc:161] P 99be4ff1114d4aaabcc537604fce54b3: Beginning new tablet copy session on tablet a7a434e2df0a4ea0b2ab5c20034033c6 from peer 4792082063bd4d58bc222c9c95b16335 at {username='slave'} at 127.23.104.201:49997: session id = 4792082063bd4d58bc222c9c95b16335-a7a434e2df0a4ea0b2ab5c20034033c6
I20240503 02:15:07.812527 8972 tablet_copy_source_session.cc:205] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:07.813681 12218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a434e2df0a4ea0b2ab5c20034033c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:07.816784 11510 raft_consensus.cc:1050] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d: attempting to promote NON_VOTER 4792082063bd4d58bc222c9c95b16335 to VOTER
I20240503 02:15:07.817277 11999 raft_consensus.cc:1050] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431: attempting to promote NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df to VOTER
I20240503 02:15:07.817975 11510 consensus_queue.cc:237] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:07.818399 11999 consensus_queue.cc:237] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:07.820997 12218 tablet_copy_client.cc:783] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:07.821341 12218 tablet_copy_client.cc:647] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:07.823151 10816 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:07.823587 8690 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:07.824904 11561 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:07.825214 12226 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:07.825937 8425 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:07.826964 9214 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:07.827031 11561 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:07.830016 12227 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:07.840884 9083 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:07.842523 8950 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:07.843292 8820 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:07.844086 12227 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:07.844455 11561 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:07.845177 12227 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:07.846055 10946 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:07.846467 8687 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:07.848415 11984 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:07.848922 11984 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:07.857410 8558 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:07.858086 12236 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:07.858523 9214 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:07.864743 12233 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:07.872562 8424 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:07.873703 11984 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:07.885375 12100 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:07.885928 12100 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335: Bootstrap complete.
I20240503 02:15:07.886245 12100 ts_tablet_manager.cc:1367] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335: Time spent bootstrapping tablet: real 0.081s user 0.053s sys 0.007s
I20240503 02:15:07.888273 12100 raft_consensus.cc:348] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:07.888590 12100 raft_consensus.cc:729] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4792082063bd4d58bc222c9c95b16335, State: Initialized, Role: LEARNER
I20240503 02:15:07.888810 12100 consensus_queue.cc:260] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:07.890743 12100 ts_tablet_manager.cc:1398] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:15:07.891582 8972 tablet_copy_service.cc:342] P 99be4ff1114d4aaabcc537604fce54b3: Request end of tablet copy session 4792082063bd4d58bc222c9c95b16335-95a8f2627c5c49a39291c96eb00fec18 received from {username='slave'} at 127.23.104.201:49997
I20240503 02:15:07.891716 8972 tablet_copy_service.cc:433] P 99be4ff1114d4aaabcc537604fce54b3: ending tablet copy session 4792082063bd4d58bc222c9c95b16335-95a8f2627c5c49a39291c96eb00fec18 on tablet 95a8f2627c5c49a39291c96eb00fec18 with peer 4792082063bd4d58bc222c9c95b16335
I20240503 02:15:07.898926 12218 tablet_copy_client.cc:523] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:07.902923 12218 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335: Bootstrap starting.
I20240503 02:15:07.909919 8950 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:07.910504 11984 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:07.916276 8819 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:07.917646 11984 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:15:07.919008 8492 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:45403, user_credentials={real_user=slave}} blocked reactor thread for 75164.9us
I20240503 02:15:07.921231 11561 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:07.924144 10815 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:07.933712 11984 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:07.936571 8557 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:07.941949 8275 catalog_manager.cc:5484] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df reported cstate change: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:07.944082 8282 catalog_manager.cc:5484] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:07.943562 8687 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:07.946138 9213 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:07.946856 9081 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:07.951570 8424 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:07.953474 10947 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:07.954126 8423 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:07.956226 8820 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:07.960484 8690 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:07.974119 8820 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:07.977977 8952 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:07.985500 8952 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:07.988197 9213 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.001859 12218 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:08.002360 12218 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335: Bootstrap complete.
I20240503 02:15:08.002689 12218 ts_tablet_manager.cc:1367] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335: Time spent bootstrapping tablet: real 0.100s user 0.052s sys 0.008s
I20240503 02:15:08.003660 12218 raft_consensus.cc:348] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:08.004101 12218 raft_consensus.cc:729] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4792082063bd4d58bc222c9c95b16335, State: Initialized, Role: LEARNER
I20240503 02:15:08.004396 12218 consensus_queue.cc:260] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:08.005406 12218 ts_tablet_manager.cc:1398] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240503 02:15:08.006206 8972 tablet_copy_service.cc:342] P 99be4ff1114d4aaabcc537604fce54b3: Request end of tablet copy session 4792082063bd4d58bc222c9c95b16335-a7a434e2df0a4ea0b2ab5c20034033c6 received from {username='slave'} at 127.23.104.201:49997
I20240503 02:15:08.006325 8972 tablet_copy_service.cc:433] P 99be4ff1114d4aaabcc537604fce54b3: ending tablet copy session 4792082063bd4d58bc222c9c95b16335-a7a434e2df0a4ea0b2ab5c20034033c6 on tablet a7a434e2df0a4ea0b2ab5c20034033c6 with peer 4792082063bd4d58bc222c9c95b16335
I20240503 02:15:08.022385 9081 consensus_queue.cc:237] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:08.025430 8557 consensus_queue.cc:237] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:08.028219 10816 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.028273 9083 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.028812 8424 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.028889 8687 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.028831 12232 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.029615 9214 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.029742 12232 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.030519 12226 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.031015 12227 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.032155 8820 raft_consensus.cc:1261] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.032650 12226 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.033145 12226 consensus_queue.cc:1035] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.035861 11561 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.040361 8269 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:08.040612 10816 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.041041 9214 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.045264 9083 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.045524 8820 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.047456 8819 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.048008 11510 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.048847 8690 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.049434 8557 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.049530 10946 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.049906 11510 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.050006 8423 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.050386 8424 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.051858 11984 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.052392 11984 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.052847 11984 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.055238 9214 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.056018 12233 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.057562 11510 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.058931 8278 catalog_manager.cc:5484] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.060691 8557 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.062232 10946 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.063907 9214 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.065430 8819 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.066412 8687 raft_consensus.cc:2932] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.070605 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b71357f9f8ef449189f485b3246869f1 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:08.070971 8276 catalog_manager.cc:5484] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 reported cstate change: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.071456 8424 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.073640 8689 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.080078 8931 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0464bacc4a4bd39dde43ccf4d9a0fc with delete_type TABLET_DATA_TOMBSTONED (TS 99be4ff1114d4aaabcc537604fce54b3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52056
I20240503 02:15:08.080236 8930 tablet_service.cc:1514] Processing DeleteTablet for tablet b71357f9f8ef449189f485b3246869f1 with delete_type TABLET_DATA_TOMBSTONED (TS 99be4ff1114d4aaabcc537604fce54b3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52056
I20240503 02:15:08.080806 12256 tablet_replica.cc:330] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3: stopping tablet replica
I20240503 02:15:08.081498 12256 raft_consensus.cc:2229] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:08.081828 12256 raft_consensus.cc:2258] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:08.083056 12256 ts_tablet_manager.cc:1875] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:08.090937 12256 ts_tablet_manager.cc:1888] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:08.091048 12256 log.cc:1196] T ee0464bacc4a4bd39dde43ccf4d9a0fc P 99be4ff1114d4aaabcc537604fce54b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/wals/ee0464bacc4a4bd39dde43ccf4d9a0fc
I20240503 02:15:08.091892 12256 tablet_replica.cc:330] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3: stopping tablet replica
I20240503 02:15:08.092154 12256 raft_consensus.cc:2229] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:08.092521 12256 raft_consensus.cc:2258] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:08.092202 8268 catalog_manager.cc:4830] TS 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): tablet ee0464bacc4a4bd39dde43ccf4d9a0fc (table rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]) successfully deleted
I20240503 02:15:08.096648 12256 ts_tablet_manager.cc:1875] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:08.106006 12256 ts_tablet_manager.cc:1888] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:08.106145 12256 log.cc:1196] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/wals/b71357f9f8ef449189f485b3246869f1
I20240503 02:15:08.106835 8268 catalog_manager.cc:4830] TS 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): tablet b71357f9f8ef449189f485b3246869f1 (table rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]) successfully deleted
I20240503 02:15:08.112118 12192 raft_consensus.cc:1050] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c: attempting to promote NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df to VOTER
I20240503 02:15:08.113381 12192 consensus_queue.cc:237] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:08.117195 10816 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.117354 8687 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.118059 12036 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.118649 12036 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.120949 8557 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.122087 12191 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.146989 9083 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.147611 8819 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.148128 8949 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.149336 12268 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.149822 12192 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.150713 12036 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.155587 12036 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.157552 9214 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.158154 12268 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.157665 9083 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.161520 10816 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.162755 8687 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.163450 8949 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.168102 8277 catalog_manager.cc:5484] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df reported cstate change: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.169814 8557 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.182133 8819 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.185905 9214 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.209206 10946 raft_consensus.cc:1203] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:08.256460 10946 raft_consensus.cc:1203] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:08.290611 12048 raft_consensus.cc:1050] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf: attempting to promote NON_VOTER 4792082063bd4d58bc222c9c95b16335 to VOTER
I20240503 02:15:08.291370 12048 consensus_queue.cc:237] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:08.294704 10946 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.295569 8687 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.296180 9083 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.297116 8949 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.297562 8423 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.298048 8557 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.298919 9214 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.299067 12296 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.301092 12297 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.301466 12300 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.301720 12296 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.302085 12293 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.302690 12298 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.313103 12299 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.317821 12296 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.322024 8689 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.322655 8557 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.322986 9213 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.323652 8423 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.329802 8949 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.333055 8278 catalog_manager.cc:5484] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79e41f25066e4e5abbd71e48d545c6bf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.334352 10946 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.337122 9083 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.337996 8819 consensus_queue.cc:237] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:08.341440 8423 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.341454 9213 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.341964 12296 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.342201 12300 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.342502 8689 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.342947 8557 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.343000 12300 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.343713 12300 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.347337 10947 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.347033 12300 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.348367 12296 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.349515 9081 raft_consensus.cc:1261] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.349960 10947 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.352478 12297 consensus_queue.cc:1035] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.352989 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3bca5fd721f84f38bff08308965eb33f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:08.353924 8273 catalog_manager.cc:5484] T 3bca5fd721f84f38bff08308965eb33f P 79e41f25066e4e5abbd71e48d545c6bf reported cstate change: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New cstate: current_term: 1 leader_uuid: "79e41f25066e4e5abbd71e48d545c6bf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:08.354236 9081 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.354828 9213 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.358454 8931 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bca5fd721f84f38bff08308965eb33f with delete_type TABLET_DATA_TOMBSTONED (TS 99be4ff1114d4aaabcc537604fce54b3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52056
I20240503 02:15:08.358841 12256 tablet_replica.cc:330] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3: stopping tablet replica
I20240503 02:15:08.359033 12256 raft_consensus.cc:2229] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:08.359445 12256 raft_consensus.cc:2258] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:08.361105 12256 ts_tablet_manager.cc:1875] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:08.361442 8689 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.364167 8423 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.364607 8558 raft_consensus.cc:2932] T 3bca5fd721f84f38bff08308965eb33f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.366678 12256 ts_tablet_manager.cc:1888] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:08.366791 12256 log.cc:1196] T 3bca5fd721f84f38bff08308965eb33f P 99be4ff1114d4aaabcc537604fce54b3: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-4/wal/wals/3bca5fd721f84f38bff08308965eb33f
I20240503 02:15:08.367566 8268 catalog_manager.cc:4830] TS 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): tablet 3bca5fd721f84f38bff08308965eb33f (table rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]) successfully deleted
I20240503 02:15:08.592842 8423 consensus_queue.cc:237] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:08.595623 9081 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.595984 8949 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.596160 10816 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.596544 8689 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.596949 12192 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.597240 12264 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.597573 8557 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.598037 12191 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.598160 9214 raft_consensus.cc:1261] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 4c668b2e2d5c4edba46cbc5c36065c0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.598901 12191 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.599136 12192 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.599614 12192 consensus_queue.cc:1035] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.602106 12264 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.602746 8557 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.603250 10816 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.604274 8689 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.607221 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1ea7444883c642419cd3e509f2b8d090 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:08.607679 9081 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.609540 8277 catalog_manager.cc:5484] T 1ea7444883c642419cd3e509f2b8d090 P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:15:08.610025 9214 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.610308 8949 raft_consensus.cc:2932] T 1ea7444883c642419cd3e509f2b8d090 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:08.614413 8801 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea7444883c642419cd3e509f2b8d090 with delete_type TABLET_DATA_TOMBSTONED (TS 79e41f25066e4e5abbd71e48d545c6bf not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38550
I20240503 02:15:08.615713 12328 tablet_replica.cc:330] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf: stopping tablet replica
I20240503 02:15:08.616012 12328 raft_consensus.cc:2229] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:08.616454 12328 raft_consensus.cc:2258] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:08.617698 12328 ts_tablet_manager.cc:1875] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:08.624112 12328 ts_tablet_manager.cc:1888] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:08.624301 12328 log.cc:1196] T 1ea7444883c642419cd3e509f2b8d090 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/1ea7444883c642419cd3e509f2b8d090
I20240503 02:15:08.625025 8267 catalog_manager.cc:4830] TS 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747): tablet 1ea7444883c642419cd3e509f2b8d090 (table rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]) successfully deleted
I20240503 02:15:08.739113 11676 raft_consensus.cc:1050] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3: attempting to promote NON_VOTER 4792082063bd4d58bc222c9c95b16335 to VOTER
I20240503 02:15:08.740024 11676 consensus_queue.cc:237] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:08.742801 9214 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.743017 9081 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.743146 8819 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.743445 10947 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.743892 12151 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.744365 8423 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.744704 12151 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.744951 8689 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.747361 8557 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.748374 11918 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.748953 12350 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.749528 11918 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.750164 11918 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.751400 12350 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.777365 12151 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.785046 8423 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.792308 8558 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.795071 9081 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.795215 10947 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.795357 8689 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.800113 8819 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.799324 8277 catalog_manager.cc:5484] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:08.809959 8949 consensus_queue.cc:237] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:08.811985 9214 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.814687 10947 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.814678 9083 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.815428 12356 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.815629 8424 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.815733 8689 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.816144 12348 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.816890 12348 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.817346 12151 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.820019 8558 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.820814 12346 raft_consensus.cc:1050] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3: attempting to promote NON_VOTER 4792082063bd4d58bc222c9c95b16335 to VOTER
I20240503 02:15:08.820621 11676 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.822146 11918 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.822001 12346 consensus_queue.cc:237] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:08.823879 8424 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.824549 9213 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.825595 8268 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 95a8f2627c5c49a39291c96eb00fec18 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:08.827592 8689 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.828707 9083 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.828727 8277 catalog_manager.cc:5484] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:08.833078 10947 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.834380 8801 tablet_service.cc:1514] Processing DeleteTablet for tablet 95a8f2627c5c49a39291c96eb00fec18 with delete_type TABLET_DATA_TOMBSTONED (TS 79e41f25066e4e5abbd71e48d545c6bf not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38550
I20240503 02:15:08.834882 12328 tablet_replica.cc:330] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf: stopping tablet replica
I20240503 02:15:08.834846 12350 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.835085 12328 raft_consensus.cc:2229] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:08.835472 12328 raft_consensus.cc:2258] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:08.836701 9213 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.836786 8558 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.838367 12328 ts_tablet_manager.cc:1875] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:08.846475 10947 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.847399 8819 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.847538 9083 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.848489 8690 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.848506 12328 ts_tablet_manager.cc:1888] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:08.848621 12328 log.cc:1196] T 95a8f2627c5c49a39291c96eb00fec18 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/95a8f2627c5c49a39291c96eb00fec18
I20240503 02:15:08.849117 8558 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.849814 12356 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.849995 8267 catalog_manager.cc:4830] TS 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747): tablet 95a8f2627c5c49a39291c96eb00fec18 (table rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]) successfully deleted
I20240503 02:15:08.850708 9213 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.851572 8424 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:08.851647 12356 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.852134 12357 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.852595 11918 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.852840 12356 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.853071 11676 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.853703 12348 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:08.871598 12348 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.872491 10947 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.874361 8690 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.878660 8277 catalog_manager.cc:5484] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 reported cstate change: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.879024 8425 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.882786 9083 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.883445 9213 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.883908 8558 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.887352 8819 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.891716 8949 consensus_queue.cc:237] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:08.900509 8424 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.901206 10947 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.901979 11676 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.902626 11676 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.903623 8689 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.906052 8558 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.906934 12356 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.908591 12356 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.909361 9083 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.910655 9214 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:08.910589 12356 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.911334 11676 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:08.911301 12356 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.913092 9213 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.913162 8689 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.918891 10947 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.919569 8273 catalog_manager.cc:5484] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.919611 9083 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.922017 8268 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7a434e2df0a4ea0b2ab5c20034033c6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:08.921656 8558 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.923166 8424 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:08.924275 8801 tablet_service.cc:1514] Processing DeleteTablet for tablet a7a434e2df0a4ea0b2ab5c20034033c6 with delete_type TABLET_DATA_TOMBSTONED (TS 79e41f25066e4e5abbd71e48d545c6bf not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38550
I20240503 02:15:08.924813 12328 tablet_replica.cc:330] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: stopping tablet replica
I20240503 02:15:08.925034 12328 raft_consensus.cc:2229] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:08.925434 12328 raft_consensus.cc:2258] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:08.926663 12328 ts_tablet_manager.cc:1875] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:08.936414 12328 ts_tablet_manager.cc:1888] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:08.936584 12328 log.cc:1196] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/a7a434e2df0a4ea0b2ab5c20034033c6
I20240503 02:15:08.937724 8267 catalog_manager.cc:4830] TS 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747): tablet a7a434e2df0a4ea0b2ab5c20034033c6 (table rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]) successfully deleted
I20240503 02:15:08.949514 9083 consensus_queue.cc:237] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } }
I20240503 02:15:08.952138 8949 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:08.952570 8689 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:08.952816 8819 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:08.953177 12237 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:08.953678 9213 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:08.954742 11510 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:08.955297 11510 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:08.957739 11984 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:08.971602 8558 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:08.972146 8424 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:08.972445 12237 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:08.974298 11984 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:08.974920 11984 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:08.977370 8819 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:08.979607 8689 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:08.983150 8274 catalog_manager.cc:5484] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:08.984094 8949 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:08.996050 9083 consensus_queue.cc:237] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:08.995965 9213 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
W20240503 02:15:09.006500 9019 consensus_peers.cc:485] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d -> Peer e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200:45027): Couldn't send request to peer e512cb0d8f244ac6ab0e7150a0a8e2df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5028c92b346845c38b3356a0d8c7507f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:09.006510 8690 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:09.006605 8949 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:09.006893 9213 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:09.007176 8819 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:09.007143 12237 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:09.007359 11984 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:09.007717 12374 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:09.009635 11510 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:09.014025 8424 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:09.019315 8557 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:09.020203 11510 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:09.024013 11984 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.024590 8558 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } }
I20240503 02:15:09.029353 8425 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:09.030376 8558 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.031747 8267 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5028c92b346845c38b3356a0d8c7507f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:09.032529 9213 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.035094 8819 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.036312 8949 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.041018 11984 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:09.041358 8274 catalog_manager.cc:5484] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf reported cstate change: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.046185 8689 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.050853 8425 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.304687 8558 consensus_queue.cc:237] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } }
I20240503 02:15:09.307235 8424 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:09.307432 8689 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:09.307943 8949 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:09.308209 9083 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:09.308264 12404 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:09.308450 9213 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:09.308648 12226 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:09.309216 8820 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:09.309209 12226 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:09.309516 12404 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:09.310003 12329 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:09.310524 12329 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:09.324471 12226 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:09.325604 8952 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:09.325853 9083 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:09.329352 8424 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:09.334935 8273 catalog_manager.cc:5484] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:09.335106 8820 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:09.341768 8689 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:09.351806 8558 consensus_queue.cc:237] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:09.355226 8949 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:09.355546 9083 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:09.355927 8820 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:09.356434 8424 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:09.356375 12329 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:09.350987 9213 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } }
I20240503 02:15:09.358592 12404 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:09.358937 12329 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:09.362715 12404 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:09.364420 12404 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.365221 9214 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:09.367190 12329 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:09.368448 8424 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.368609 9214 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.373298 9083 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.374903 8277 catalog_manager.cc:5484] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.375701 8269 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4679ab055c1a40f19bf7b602451044f6 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:15:09.378825 8690 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:09.380393 12411 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:15:09.381332 8493 consensus_peers.cc:485] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 -> Peer e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200:45027): Couldn't send request to peer e512cb0d8f244ac6ab0e7150a0a8e2df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4679ab055c1a40f19bf7b602451044f6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:09.382510 8690 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.382378 8820 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.385104 8949 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:09.594111 12424 ts_tablet_manager.cc:902] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df: Initiating tablet copy from peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:15:09.595523 12424 tablet_copy_client.cc:308] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:41237
I20240503 02:15:09.596237 9103 tablet_copy_service.cc:140] P c399efcbba924863ac89d156dc07ac7d: Received BeginTabletCopySession request for tablet 5028c92b346845c38b3356a0d8c7507f from peer e512cb0d8f244ac6ab0e7150a0a8e2df ({username='slave'} at 127.23.104.200:52887)
I20240503 02:15:09.596378 9103 tablet_copy_service.cc:161] P c399efcbba924863ac89d156dc07ac7d: Beginning new tablet copy session on tablet 5028c92b346845c38b3356a0d8c7507f from peer e512cb0d8f244ac6ab0e7150a0a8e2df at {username='slave'} at 127.23.104.200:52887: session id = e512cb0d8f244ac6ab0e7150a0a8e2df-5028c92b346845c38b3356a0d8c7507f
I20240503 02:15:09.598986 9103 tablet_copy_source_session.cc:205] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:09.600281 12424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5028c92b346845c38b3356a0d8c7507f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:09.604593 12424 tablet_copy_client.cc:783] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:09.604910 12424 tablet_copy_client.cc:647] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:09.606424 12424 tablet_copy_client.cc:523] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:09.609055 12424 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap starting.
I20240503 02:15:09.646324 12424 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:09.646771 12424 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap complete.
I20240503 02:15:09.647033 12424 ts_tablet_manager.cc:1367] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent bootstrapping tablet: real 0.038s user 0.034s sys 0.000s
I20240503 02:15:09.647773 12424 raft_consensus.cc:348] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:09.648049 12424 raft_consensus.cc:729] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e512cb0d8f244ac6ab0e7150a0a8e2df, State: Initialized, Role: LEARNER
I20240503 02:15:09.648262 12424 consensus_queue.cc:260] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:09.648986 12424 ts_tablet_manager.cc:1398] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 02:15:09.649609 9103 tablet_copy_service.cc:342] P c399efcbba924863ac89d156dc07ac7d: Request end of tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-5028c92b346845c38b3356a0d8c7507f received from {username='slave'} at 127.23.104.200:52887
I20240503 02:15:09.649740 9103 tablet_copy_service.cc:433] P c399efcbba924863ac89d156dc07ac7d: ending tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-5028c92b346845c38b3356a0d8c7507f on tablet 5028c92b346845c38b3356a0d8c7507f with peer e512cb0d8f244ac6ab0e7150a0a8e2df
I20240503 02:15:09.820025 12424 ts_tablet_manager.cc:902] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Initiating tablet copy from peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923)
I20240503 02:15:09.821414 12424 tablet_copy_client.cc:308] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:45923
I20240503 02:15:09.822099 8578 tablet_copy_service.cc:140] P 2a1893ab239f4163a06bdbf8ce579431: Received BeginTabletCopySession request for tablet 4679ab055c1a40f19bf7b602451044f6 from peer e512cb0d8f244ac6ab0e7150a0a8e2df ({username='slave'} at 127.23.104.200:41677)
I20240503 02:15:09.822273 8578 tablet_copy_service.cc:161] P 2a1893ab239f4163a06bdbf8ce579431: Beginning new tablet copy session on tablet 4679ab055c1a40f19bf7b602451044f6 from peer e512cb0d8f244ac6ab0e7150a0a8e2df at {username='slave'} at 127.23.104.200:41677: session id = e512cb0d8f244ac6ab0e7150a0a8e2df-4679ab055c1a40f19bf7b602451044f6
I20240503 02:15:09.824712 8578 tablet_copy_source_session.cc:205] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:09.826295 12424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4679ab055c1a40f19bf7b602451044f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:09.832438 12424 tablet_copy_client.cc:783] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:09.832646 12424 tablet_copy_client.cc:647] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:09.834041 12424 tablet_copy_client.cc:523] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:09.836848 12424 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap starting.
I20240503 02:15:09.877952 12424 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:09.878325 12424 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap complete.
I20240503 02:15:09.878547 12424 ts_tablet_manager.cc:1367] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent bootstrapping tablet: real 0.042s user 0.033s sys 0.004s
I20240503 02:15:09.879175 12424 raft_consensus.cc:348] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:09.879406 12424 raft_consensus.cc:729] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e512cb0d8f244ac6ab0e7150a0a8e2df, State: Initialized, Role: LEARNER
I20240503 02:15:09.879603 12424 consensus_queue.cc:260] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:09.880215 12424 ts_tablet_manager.cc:1398] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:15:09.880993 8578 tablet_copy_service.cc:342] P 2a1893ab239f4163a06bdbf8ce579431: Request end of tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-4679ab055c1a40f19bf7b602451044f6 received from {username='slave'} at 127.23.104.200:41677
I20240503 02:15:09.881129 8578 tablet_copy_service.cc:433] P 2a1893ab239f4163a06bdbf8ce579431: ending tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-4679ab055c1a40f19bf7b602451044f6 on tablet 4679ab055c1a40f19bf7b602451044f6 with peer e512cb0d8f244ac6ab0e7150a0a8e2df
I20240503 02:15:10.032184 10816 raft_consensus.cc:1203] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:10.226945 10816 raft_consensus.cc:1203] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:15:10.598526 12457 raft_consensus.cc:1050] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d: attempting to promote NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df to VOTER
I20240503 02:15:10.599614 12457 consensus_queue.cc:237] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:10.602329 9213 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:10.602720 8820 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:10.602929 12489 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:10.603324 8949 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:10.604827 12411 raft_consensus.cc:1050] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431: attempting to promote NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df to VOTER
I20240503 02:15:10.605024 12489 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:10.605764 12489 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:10.606007 12411 consensus_queue.cc:237] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:10.611608 8690 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:10.611606 10816 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:10.612291 12491 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:10.612797 12489 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:10.618319 8425 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:10.618899 12457 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:10.625780 8557 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:10.625306 12489 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.626025 10816 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:10.626436 12457 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:10.626644 12420 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:10.628319 8557 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.630726 8949 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.633932 10816 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.634946 8425 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.636104 9213 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.636451 8277 catalog_manager.cc:5484] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.631523 8819 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.646221 9213 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:10.646366 8424 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:10.646687 12420 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:10.648660 8689 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:10.650676 9081 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:10.650947 8820 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:10.651232 12329 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:10.651564 12411 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:10.652213 12329 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:10.652626 12420 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:10.653012 8689 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.659200 8949 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:10.660197 12329 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:10.672307 12329 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.677618 8952 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.686524 8687 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.688483 9213 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.689352 8425 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.692373 9081 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.693272 8820 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.694092 10815 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.710772 8282 catalog_manager.cc:5484] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 5 to 6, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:15:10.717561 8557 consensus_queue.cc:237] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:10.720252 10815 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:10.721114 8687 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:10.721074 12329 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:10.722858 12329 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:10.725464 8425 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:10.725469 8949 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:10.726334 12411 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:10.726667 9081 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:10.727130 12411 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:10.727516 12507 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:10.729693 9214 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:10.730325 12507 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:10.732460 12507 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.732816 8687 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.737396 9214 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.739012 8425 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.739298 8277 catalog_manager.cc:5484] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 reported cstate change: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.742535 9081 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.746227 8801 tablet_service.cc:1514] Processing DeleteTablet for tablet 4679ab055c1a40f19bf7b602451044f6 with delete_type TABLET_DATA_TOMBSTONED (TS 79e41f25066e4e5abbd71e48d545c6bf not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38550
I20240503 02:15:10.748221 12522 tablet_replica.cc:330] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: stopping tablet replica
I20240503 02:15:10.748453 12522 raft_consensus.cc:2229] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:10.748857 12522 raft_consensus.cc:2258] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:10.749935 12522 ts_tablet_manager.cc:1875] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:10.752017 8269 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4679ab055c1a40f19bf7b602451044f6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:10.752775 10815 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:10.755923 12522 ts_tablet_manager.cc:1888] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:10.756034 12522 log.cc:1196] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/4679ab055c1a40f19bf7b602451044f6
I20240503 02:15:10.756983 8267 catalog_manager.cc:4830] TS 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747): tablet 4679ab055c1a40f19bf7b602451044f6 (table rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]) successfully deleted
I20240503 02:15:10.758674 8949 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:11.100430 9081 consensus_queue.cc:237] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:11.102703 9214 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:11.102857 8949 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:11.102866 10815 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:11.103296 12489 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:11.103675 8557 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:11.103670 8687 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:11.103689 8423 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:11.103893 12492 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:11.104315 12497 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:11.104792 12492 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:11.105090 12497 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:11.107878 12488 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:11.108580 8949 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:11.111304 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5028c92b346845c38b3356a0d8c7507f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:11.111569 8557 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:11.111562 8687 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:11.112215 9214 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:11.114207 8277 catalog_manager.cc:5484] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:11.116487 12489 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:11.118203 10815 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:11.122418 8423 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:11.124172 8801 tablet_service.cc:1514] Processing DeleteTablet for tablet 5028c92b346845c38b3356a0d8c7507f with delete_type TABLET_DATA_TOMBSTONED (TS 79e41f25066e4e5abbd71e48d545c6bf not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38550
I20240503 02:15:11.124604 12522 tablet_replica.cc:330] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf: stopping tablet replica
I20240503 02:15:11.124832 12522 raft_consensus.cc:2229] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:11.125280 12522 raft_consensus.cc:2258] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:11.126350 12522 ts_tablet_manager.cc:1875] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:11.131280 12522 ts_tablet_manager.cc:1888] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:11.131376 12522 log.cc:1196] T 5028c92b346845c38b3356a0d8c7507f P 79e41f25066e4e5abbd71e48d545c6bf: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-3/wal/wals/5028c92b346845c38b3356a0d8c7507f
I20240503 02:15:11.135408 8267 catalog_manager.cc:4830] TS 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747): tablet 5028c92b346845c38b3356a0d8c7507f (table rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]) successfully deleted
I20240503 02:15:11.559932 8537 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:11.594113 8403 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:11.595837 10927 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:11.601874 8670 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:11.613054 8931 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:15:11.639576 11059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:11.640635 9062 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:11.643962 8801 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:15:11.669835 10796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:11.718601 9193 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:15:11.721661 9019 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.194:45923, user_credentials={real_user=slave}} blocked reactor thread for 63983.5us
I20240503 02:15:11.937901 8537 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:11.945533 10927 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:11.952867 8403 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:11.971089 8801 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:15:11.979005 8670 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:12.019366 8931 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:15:12.035909 9193 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:15:12.040452 9062 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:12.064514 11059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:12.066995 10796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:12.201722 9081 consensus_queue.cc:237] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:12.204645 10815 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.205209 8557 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.205271 8687 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.205533 9214 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.205765 12488 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.205986 8423 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.206359 12488 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.206859 12488 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.208242 12585 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.208880 12488 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.239930 12488 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.247969 10816 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.256660 8687 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.281847 8274 catalog_manager.cc:5484] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df reported cstate change: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.296191 8819 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.308101 12585 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:15:12.308562 9018 consensus_peers.cc:485] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d -> Peer dabeea74472746538132a520710248ea (127.23.104.202:37759): Couldn't send request to peer dabeea74472746538132a520710248ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df68cc6032b14cae9e1cac074976baf8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:12.309407 9214 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.309856 8557 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.317411 8423 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.327622 8819 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.334024 9081 consensus_queue.cc:237] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:12.339407 9214 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.340020 8952 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.340181 12488 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.340571 8687 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.340849 12488 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.341210 10816 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.341400 12488 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.342372 12585 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:15:12.344031 9017 consensus_peers.cc:485] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d -> Peer 4792082063bd4d58bc222c9c95b16335 (127.23.104.201:46877): Couldn't send request to peer 4792082063bd4d58bc222c9c95b16335. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5028c92b346845c38b3356a0d8c7507f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:12.344624 8557 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.344786 8423 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.345109 12647 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.345758 12647 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20240503 02:15:12.349372 12585 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.349943 10816 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.355469 8423 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.356475 8952 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.357216 8557 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.359083 9213 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.361505 8273 catalog_manager.cc:5484] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df reported cstate change: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.365715 8690 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.382822 9081 consensus_queue.cc:237] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
W20240503 02:15:12.385700 9019 consensus_peers.cc:485] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d -> Peer e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200:45027): Couldn't send request to peer e512cb0d8f244ac6ab0e7150a0a8e2df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b71357f9f8ef449189f485b3246869f1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:12.385890 9214 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.385942 8557 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.385993 10947 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.386528 12647 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.386669 8423 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.386953 12632 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.387164 8819 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.387430 12648 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.387806 12632 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.388239 8690 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.388314 12647 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.390523 12632 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.405025 12646 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.405692 9214 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.406140 8821 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.410727 10947 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.411087 8557 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.417090 8277 catalog_manager.cc:5484] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.417658 8690 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.427902 8425 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.443348 8952 consensus_queue.cc:237] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:12.446285 11079 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.447124 9214 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.447237 12504 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.447679 12350 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:15:12.448516 8886 consensus_peers.cc:485] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 -> Peer 4792082063bd4d58bc222c9c95b16335 (127.23.104.201:46877): Couldn't send request to peer 4792082063bd4d58bc222c9c95b16335. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64c52f9977ae4eb4ac803be99b6426c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:12.448720 8690 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.449327 9083 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.450608 12350 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.450938 12504 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.459412 8557 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.460191 12680 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.461313 8425 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.461902 12680 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.467254 12504 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.469660 11079 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.478248 9214 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.480780 9081 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.490906 8557 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.491353 8425 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.495988 8277 catalog_manager.cc:5484] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.510967 8690 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.539237 9214 consensus_queue.cc:237] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:12.543044 10816 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.543992 8690 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.545043 8557 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.545553 8425 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.545801 9083 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.546223 12533 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.546864 12533 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.547398 12533 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.548141 12533 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.550055 12653 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.571835 12533 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.576670 10815 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.580220 8952 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.579982 8690 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.580888 12653 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.588215 8273 catalog_manager.cc:5484] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df reported cstate change: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.584334 8557 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.594810 9083 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.598770 8423 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.606859 8952 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
W20240503 02:15:12.613576 9149 consensus_peers.cc:485] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 -> Peer dabeea74472746538132a520710248ea (127.23.104.202:37759): Couldn't send request to peer dabeea74472746538132a520710248ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 879167ff15db41a7b4ef21fa10c44b27. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:12.620736 8952 consensus_queue.cc:237] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
W20240503 02:15:12.624730 8888 consensus_peers.cc:485] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 -> Peer e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200:45027): Couldn't send request to peer e512cb0d8f244ac6ab0e7150a0a8e2df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7a434e2df0a4ea0b2ab5c20034033c6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:12.625036 10947 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.625105 8425 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.625120 8557 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.625597 9214 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.625689 12680 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.626041 12711 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.626147 9083 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.626258 12504 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.626590 12350 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.626719 8690 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.626858 12504 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.627300 12687 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.632844 12680 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.633430 9083 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.633705 8690 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.637717 9214 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.641486 8273 catalog_manager.cc:5484] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 reported cstate change: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.649061 8425 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.648607 8557 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.657843 10947 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.675081 8952 consensus_queue.cc:237] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:12.677995 8690 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.678498 8425 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.678906 9214 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.679126 12504 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.679450 12711 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.680925 10815 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.681816 9083 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.682488 12687 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:15:12.684343 8886 consensus_peers.cc:485] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 -> Peer 4792082063bd4d58bc222c9c95b16335 (127.23.104.201:46877): Couldn't send request to peer 4792082063bd4d58bc222c9c95b16335. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c28f0e30cfe46fda64dd2b0fd7847c6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:12.686780 12687 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.687425 12687 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.688992 8557 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.689742 12687 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.710690 12711 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.711759 9213 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.714443 8423 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.716305 9083 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.716480 8277 catalog_manager.cc:5484] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:12.716714 8558 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.748333 8687 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.750777 10815 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:12.807579 8952 consensus_queue.cc:237] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:12.810654 10947 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.811188 9083 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.811352 9213 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 02:15:12.811750 8887 consensus_peers.cc:485] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 -> Peer dabeea74472746538132a520710248ea (127.23.104.202:37759): Couldn't send request to peer dabeea74472746538132a520710248ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95a8f2627c5c49a39291c96eb00fec18. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:12.811924 12724 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.812491 12724 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.812927 8423 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.813058 12724 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.813761 8690 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.814185 12711 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.815284 12724 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.817247 8558 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.817950 12680 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.844481 12724 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.845330 9213 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.847980 10947 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.850425 8690 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.852357 8279 catalog_manager.cc:5484] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.852995 12737 ts_tablet_manager.cc:902] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea: Initiating tablet copy from peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:15:12.854558 8558 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.859949 9083 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.862859 8423 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:12.879827 12737 tablet_copy_client.cc:308] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:41237
I20240503 02:15:12.890841 9103 tablet_copy_service.cc:140] P c399efcbba924863ac89d156dc07ac7d: Received BeginTabletCopySession request for tablet df68cc6032b14cae9e1cac074976baf8 from peer dabeea74472746538132a520710248ea ({username='slave'} at 127.23.104.202:46779)
I20240503 02:15:12.891027 9103 tablet_copy_service.cc:161] P c399efcbba924863ac89d156dc07ac7d: Beginning new tablet copy session on tablet df68cc6032b14cae9e1cac074976baf8 from peer dabeea74472746538132a520710248ea at {username='slave'} at 127.23.104.202:46779: session id = dabeea74472746538132a520710248ea-df68cc6032b14cae9e1cac074976baf8
I20240503 02:15:12.893358 9103 tablet_copy_source_session.cc:205] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:12.894630 12737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df68cc6032b14cae9e1cac074976baf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:12.898484 12756 ts_tablet_manager.cc:902] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335: Initiating tablet copy from peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:15:12.901001 8558 consensus_queue.cc:237] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:12.902320 12737 tablet_copy_client.cc:783] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:12.902643 12737 tablet_copy_client.cc:647] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:12.903038 12756 tablet_copy_client.cc:308] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:41237
I20240503 02:15:12.903723 9103 tablet_copy_service.cc:140] P c399efcbba924863ac89d156dc07ac7d: Received BeginTabletCopySession request for tablet 5028c92b346845c38b3356a0d8c7507f from peer 4792082063bd4d58bc222c9c95b16335 ({username='slave'} at 127.23.104.201:41501)
I20240503 02:15:12.903918 9103 tablet_copy_service.cc:161] P c399efcbba924863ac89d156dc07ac7d: Beginning new tablet copy session on tablet 5028c92b346845c38b3356a0d8c7507f from peer 4792082063bd4d58bc222c9c95b16335 at {username='slave'} at 127.23.104.201:41501: session id = 4792082063bd4d58bc222c9c95b16335-5028c92b346845c38b3356a0d8c7507f
I20240503 02:15:12.905342 8690 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 02:15:12.905493 8493 consensus_peers.cc:485] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 -> Peer e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200:45027): Couldn't send request to peer e512cb0d8f244ac6ab0e7150a0a8e2df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99e42dca2db54de5a9417df20242e2c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:12.905871 12737 tablet_copy_client.cc:523] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:12.906095 8423 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.906492 11079 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.906708 9103 tablet_copy_source_session.cc:205] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:12.907065 12507 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.907824 12507 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.908805 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5028c92b346845c38b3356a0d8c7507f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:12.909252 12515 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.909364 12737 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea: Bootstrap starting.
I20240503 02:15:12.913933 9213 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.914070 9083 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.914541 12515 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.915745 8952 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:12.915709 12507 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.917551 12507 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:12.918192 12507 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.918705 9213 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.925392 12756 tablet_copy_client.cc:783] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:12.925685 12756 tablet_copy_client.cc:647] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:12.927134 8277 catalog_manager.cc:5484] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.934316 12756 tablet_copy_client.cc:523] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:12.938681 12756 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335: Bootstrap starting.
I20240503 02:15:12.979552 12737 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:12.979737 12771 ts_tablet_manager.cc:902] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335: Initiating tablet copy from peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403)
I20240503 02:15:12.980087 12737 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea: Bootstrap complete.
I20240503 02:15:12.980402 12737 ts_tablet_manager.cc:1367] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea: Time spent bootstrapping tablet: real 0.071s user 0.057s sys 0.010s
I20240503 02:15:12.981215 12737 raft_consensus.cc:348] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:12.981590 12737 raft_consensus.cc:729] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dabeea74472746538132a520710248ea, State: Initialized, Role: LEARNER
I20240503 02:15:12.981807 12737 consensus_queue.cc:260] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:12.981778 8949 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.982380 12737 ts_tablet_manager.cc:1398] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:15:12.983093 9103 tablet_copy_service.cc:342] P c399efcbba924863ac89d156dc07ac7d: Request end of tablet copy session dabeea74472746538132a520710248ea-df68cc6032b14cae9e1cac074976baf8 received from {username='slave'} at 127.23.104.202:46779
I20240503 02:15:12.983225 9103 tablet_copy_service.cc:433] P c399efcbba924863ac89d156dc07ac7d: ending tablet copy session dabeea74472746538132a520710248ea-df68cc6032b14cae9e1cac074976baf8 on tablet df68cc6032b14cae9e1cac074976baf8 with peer dabeea74472746538132a520710248ea
I20240503 02:15:12.987223 12737 ts_tablet_manager.cc:902] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea: Initiating tablet copy from peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:15:12.992651 9083 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:12.993625 12770 ts_tablet_manager.cc:902] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df: Initiating tablet copy from peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:15:13.009397 11079 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:13.009362 8423 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:13.008608 8690 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:13.030716 12770 tablet_copy_client.cc:308] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:41237
I20240503 02:15:13.031643 9103 tablet_copy_service.cc:140] P c399efcbba924863ac89d156dc07ac7d: Received BeginTabletCopySession request for tablet b71357f9f8ef449189f485b3246869f1 from peer e512cb0d8f244ac6ab0e7150a0a8e2df ({username='slave'} at 127.23.104.200:52887)
I20240503 02:15:13.031893 9103 tablet_copy_service.cc:161] P c399efcbba924863ac89d156dc07ac7d: Beginning new tablet copy session on tablet b71357f9f8ef449189f485b3246869f1 from peer e512cb0d8f244ac6ab0e7150a0a8e2df at {username='slave'} at 127.23.104.200:52887: session id = e512cb0d8f244ac6ab0e7150a0a8e2df-b71357f9f8ef449189f485b3246869f1
I20240503 02:15:13.035684 9103 tablet_copy_source_session.cc:205] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:13.037626 12771 tablet_copy_client.cc:308] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.197:45403
I20240503 02:15:13.038401 8972 tablet_copy_service.cc:140] P 99be4ff1114d4aaabcc537604fce54b3: Received BeginTabletCopySession request for tablet 64c52f9977ae4eb4ac803be99b6426c2 from peer 4792082063bd4d58bc222c9c95b16335 ({username='slave'} at 127.23.104.201:49997)
I20240503 02:15:13.038560 8972 tablet_copy_service.cc:161] P 99be4ff1114d4aaabcc537604fce54b3: Beginning new tablet copy session on tablet 64c52f9977ae4eb4ac803be99b6426c2 from peer 4792082063bd4d58bc222c9c95b16335 at {username='slave'} at 127.23.104.201:49997: session id = 4792082063bd4d58bc222c9c95b16335-64c52f9977ae4eb4ac803be99b6426c2
I20240503 02:15:13.041996 8972 tablet_copy_source_session.cc:205] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:13.044214 12771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c52f9977ae4eb4ac803be99b6426c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:13.044606 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71357f9f8ef449189f485b3246869f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:13.064119 12770 tablet_copy_client.cc:783] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:13.064370 12771 tablet_copy_client.cc:783] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:13.064400 12770 tablet_copy_client.cc:647] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:13.064800 12771 tablet_copy_client.cc:647] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:13.070955 12771 tablet_copy_client.cc:523] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:13.074664 12771 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335: Bootstrap starting.
I20240503 02:15:13.077822 12770 tablet_copy_client.cc:523] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:13.082046 8690 consensus_queue.cc:237] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:13.084427 12770 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap starting.
I20240503 02:15:13.104727 12737 tablet_copy_client.cc:308] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.199:32809
I20240503 02:15:13.109194 12756 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:13.109704 12756 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335: Bootstrap complete.
I20240503 02:15:13.110041 12756 ts_tablet_manager.cc:1367] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335: Time spent bootstrapping tablet: real 0.171s user 0.064s sys 0.059s
I20240503 02:15:13.111624 12756 raft_consensus.cc:348] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:13.112147 12756 raft_consensus.cc:729] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4792082063bd4d58bc222c9c95b16335, State: Initialized, Role: LEARNER
I20240503 02:15:13.112423 12756 consensus_queue.cc:260] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:13.113649 12756 ts_tablet_manager.cc:1398] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:15:13.114522 9103 tablet_copy_service.cc:342] P c399efcbba924863ac89d156dc07ac7d: Request end of tablet copy session 4792082063bd4d58bc222c9c95b16335-5028c92b346845c38b3356a0d8c7507f received from {username='slave'} at 127.23.104.201:41501
I20240503 02:15:13.114646 9103 tablet_copy_service.cc:433] P c399efcbba924863ac89d156dc07ac7d: ending tablet copy session 4792082063bd4d58bc222c9c95b16335-5028c92b346845c38b3356a0d8c7507f on tablet 5028c92b346845c38b3356a0d8c7507f with peer 4792082063bd4d58bc222c9c95b16335
I20240503 02:15:13.115816 12792 ts_tablet_manager.cc:902] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335: Initiating tablet copy from peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403)
I20240503 02:15:13.123261 9234 tablet_copy_service.cc:140] P cb90f34dc37b4b16b2a5b5a4ee012865: Received BeginTabletCopySession request for tablet 879167ff15db41a7b4ef21fa10c44b27 from peer dabeea74472746538132a520710248ea ({username='slave'} at 127.23.104.202:35173)
I20240503 02:15:13.123450 9234 tablet_copy_service.cc:161] P cb90f34dc37b4b16b2a5b5a4ee012865: Beginning new tablet copy session on tablet 879167ff15db41a7b4ef21fa10c44b27 from peer dabeea74472746538132a520710248ea at {username='slave'} at 127.23.104.202:35173: session id = dabeea74472746538132a520710248ea-879167ff15db41a7b4ef21fa10c44b27
I20240503 02:15:13.126525 9234 tablet_copy_source_session.cc:205] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:13.127993 12737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879167ff15db41a7b4ef21fa10c44b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:13.135296 12737 tablet_copy_client.cc:783] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:13.135655 12737 tablet_copy_client.cc:647] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:13.138046 11079 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.139238 11953 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.140172 12737 tablet_copy_client.cc:523] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:13.144742 12737 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea: Bootstrap starting.
I20240503 02:15:13.150291 12792 tablet_copy_client.cc:308] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.197:45403
I20240503 02:15:13.151222 8972 tablet_copy_service.cc:140] P 99be4ff1114d4aaabcc537604fce54b3: Received BeginTabletCopySession request for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 from peer 4792082063bd4d58bc222c9c95b16335 ({username='slave'} at 127.23.104.201:49997)
I20240503 02:15:13.151438 8972 tablet_copy_service.cc:161] P 99be4ff1114d4aaabcc537604fce54b3: Beginning new tablet copy session on tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 from peer 4792082063bd4d58bc222c9c95b16335 at {username='slave'} at 127.23.104.201:49997: session id = 4792082063bd4d58bc222c9c95b16335-3c28f0e30cfe46fda64dd2b0fd7847c6
I20240503 02:15:13.155445 8972 tablet_copy_source_session.cc:205] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:13.158659 8952 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.159564 8558 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.160149 12792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:13.160499 9083 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.160668 12806 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.160969 11953 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.161056 9213 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.162174 8423 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.163028 11953 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.163661 11953 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.164309 11953 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.200423 12792 tablet_copy_client.cc:783] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:13.200740 12792 tablet_copy_client.cc:647] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:13.202215 12797 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:13.203212 8558 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:13.210215 9214 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:13.211606 8952 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:13.215677 12792 tablet_copy_client.cc:523] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:13.222815 12792 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335: Bootstrap starting.
I20240503 02:15:13.222371 8279 catalog_manager.cc:5484] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:13.227733 12801 ts_tablet_manager.cc:902] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Initiating tablet copy from peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403)
W20240503 02:15:13.228682 8623 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:41237, user_credentials={real_user=slave}} blocked reactor thread for 65857.7us
I20240503 02:15:13.231222 11078 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:13.252478 9083 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
W20240503 02:15:13.288568 8623 consensus_peers.cc:485] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 -> Peer 4792082063bd4d58bc222c9c95b16335 (127.23.104.201:46877): Couldn't send request to peer 4792082063bd4d58bc222c9c95b16335. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe35d536ab6348c58e90d680383bb763. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:13.291298 11079 raft_consensus.cc:1203] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:15:13.297843 8423 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:13.306519 12824 ts_tablet_manager.cc:902] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea: Initiating tablet copy from peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403)
I20240503 02:15:13.322465 12801 tablet_copy_client.cc:308] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.197:45403
I20240503 02:15:13.323292 8972 tablet_copy_service.cc:140] P 99be4ff1114d4aaabcc537604fce54b3: Received BeginTabletCopySession request for tablet a7a434e2df0a4ea0b2ab5c20034033c6 from peer e512cb0d8f244ac6ab0e7150a0a8e2df ({username='slave'} at 127.23.104.200:41679)
I20240503 02:15:13.323446 8972 tablet_copy_service.cc:161] P 99be4ff1114d4aaabcc537604fce54b3: Beginning new tablet copy session on tablet a7a434e2df0a4ea0b2ab5c20034033c6 from peer e512cb0d8f244ac6ab0e7150a0a8e2df at {username='slave'} at 127.23.104.200:41679: session id = e512cb0d8f244ac6ab0e7150a0a8e2df-a7a434e2df0a4ea0b2ab5c20034033c6
I20240503 02:15:13.326393 8972 tablet_copy_source_session.cc:205] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:13.327862 12801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a434e2df0a4ea0b2ab5c20034033c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:13.336153 12801 tablet_copy_client.cc:783] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:13.336432 12801 tablet_copy_client.cc:647] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:13.338352 12801 tablet_copy_client.cc:523] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:13.346138 9083 consensus_queue.cc:237] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:13.352236 12792 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:13.352782 12792 tablet_bootstrap.cc:492] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335: Bootstrap complete.
I20240503 02:15:13.353185 12792 ts_tablet_manager.cc:1367] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335: Time spent bootstrapping tablet: real 0.130s user 0.059s sys 0.006s
I20240503 02:15:13.354153 12792 raft_consensus.cc:348] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:13.354668 12792 raft_consensus.cc:729] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4792082063bd4d58bc222c9c95b16335, State: Initialized, Role: LEARNER
I20240503 02:15:13.354967 12792 consensus_queue.cc:260] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:13.357771 12801 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap starting.
I20240503 02:15:13.360211 12792 ts_tablet_manager.cc:1398] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:15:13.361275 8972 tablet_copy_service.cc:342] P 99be4ff1114d4aaabcc537604fce54b3: Request end of tablet copy session 4792082063bd4d58bc222c9c95b16335-3c28f0e30cfe46fda64dd2b0fd7847c6 received from {username='slave'} at 127.23.104.201:49997
I20240503 02:15:13.361392 8972 tablet_copy_service.cc:433] P 99be4ff1114d4aaabcc537604fce54b3: ending tablet copy session 4792082063bd4d58bc222c9c95b16335-3c28f0e30cfe46fda64dd2b0fd7847c6 on tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 with peer 4792082063bd4d58bc222c9c95b16335
I20240503 02:15:13.375022 10947 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.375617 12648 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:15:13.382447 9018 consensus_peers.cc:485] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d -> Peer dabeea74472746538132a520710248ea (127.23.104.202:37759): Couldn't send request to peer dabeea74472746538132a520710248ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a30758cbd1d4845bb9eeb02e1a2942a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:13.383611 9213 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.384217 8819 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.384662 8690 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.384928 12648 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.385251 12585 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.386287 12632 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.409536 8558 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.410189 12585 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.416852 8423 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.417409 12648 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.420876 12648 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.425377 8557 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.433380 8279 catalog_manager.cc:5484] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.433746 10947 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.438318 12801 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:13.440394 12801 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap complete.
I20240503 02:15:13.440517 12771 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:13.440793 12801 ts_tablet_manager.cc:1367] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent bootstrapping tablet: real 0.083s user 0.069s sys 0.008s
I20240503 02:15:13.441102 12771 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335: Bootstrap complete.
I20240503 02:15:13.441437 12771 ts_tablet_manager.cc:1367] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335: Time spent bootstrapping tablet: real 0.367s user 0.061s sys 0.008s
I20240503 02:15:13.441829 12801 raft_consensus.cc:348] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:13.442299 12801 raft_consensus.cc:729] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e512cb0d8f244ac6ab0e7150a0a8e2df, State: Initialized, Role: LEARNER
I20240503 02:15:13.441884 8819 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.442528 12801 consensus_queue.cc:260] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:13.443186 12771 raft_consensus.cc:348] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:13.445744 12771 raft_consensus.cc:729] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4792082063bd4d58bc222c9c95b16335, State: Initialized, Role: LEARNER
I20240503 02:15:13.446029 12771 consensus_queue.cc:260] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:13.446822 12771 ts_tablet_manager.cc:1398] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:15:13.450515 8972 tablet_copy_service.cc:342] P 99be4ff1114d4aaabcc537604fce54b3: Request end of tablet copy session 4792082063bd4d58bc222c9c95b16335-64c52f9977ae4eb4ac803be99b6426c2 received from {username='slave'} at 127.23.104.201:49997
I20240503 02:15:13.450677 8972 tablet_copy_service.cc:433] P 99be4ff1114d4aaabcc537604fce54b3: ending tablet copy session 4792082063bd4d58bc222c9c95b16335-64c52f9977ae4eb4ac803be99b6426c2 on tablet 64c52f9977ae4eb4ac803be99b6426c2 with peer 4792082063bd4d58bc222c9c95b16335
I20240503 02:15:13.463927 12839 ts_tablet_manager.cc:902] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df: Initiating tablet copy from peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923)
I20240503 02:15:13.466006 12801 ts_tablet_manager.cc:1398] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent starting tablet: real 0.025s user 0.002s sys 0.021s
I20240503 02:15:13.466794 8972 tablet_copy_service.cc:342] P 99be4ff1114d4aaabcc537604fce54b3: Request end of tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-a7a434e2df0a4ea0b2ab5c20034033c6 received from {username='slave'} at 127.23.104.200:41679
I20240503 02:15:13.466946 8972 tablet_copy_service.cc:433] P 99be4ff1114d4aaabcc537604fce54b3: ending tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-a7a434e2df0a4ea0b2ab5c20034033c6 on tablet a7a434e2df0a4ea0b2ab5c20034033c6 with peer e512cb0d8f244ac6ab0e7150a0a8e2df
I20240503 02:15:13.480087 8425 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.490010 10947 raft_consensus.cc:1203] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:15:13.495126 9212 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.509788 12770 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:13.510319 12770 tablet_bootstrap.cc:492] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap complete.
I20240503 02:15:13.510655 12770 ts_tablet_manager.cc:1367] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent bootstrapping tablet: real 0.426s user 0.051s sys 0.015s
I20240503 02:15:13.511474 12770 raft_consensus.cc:348] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:13.511853 12770 raft_consensus.cc:729] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e512cb0d8f244ac6ab0e7150a0a8e2df, State: Initialized, Role: LEARNER
I20240503 02:15:13.512109 12770 consensus_queue.cc:260] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:13.512763 12770 ts_tablet_manager.cc:1398] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:15:13.516239 8690 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.521369 9103 tablet_copy_service.cc:342] P c399efcbba924863ac89d156dc07ac7d: Request end of tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-b71357f9f8ef449189f485b3246869f1 received from {username='slave'} at 127.23.104.200:52887
I20240503 02:15:13.521518 9103 tablet_copy_service.cc:433] P c399efcbba924863ac89d156dc07ac7d: ending tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-b71357f9f8ef449189f485b3246869f1 on tablet b71357f9f8ef449189f485b3246869f1 with peer e512cb0d8f244ac6ab0e7150a0a8e2df
I20240503 02:15:13.536964 12737 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:13.537461 12737 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea: Bootstrap complete.
I20240503 02:15:13.537757 12737 ts_tablet_manager.cc:1367] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea: Time spent bootstrapping tablet: real 0.393s user 0.050s sys 0.011s
I20240503 02:15:13.538889 12737 raft_consensus.cc:348] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:13.539214 12737 raft_consensus.cc:729] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dabeea74472746538132a520710248ea, State: Initialized, Role: LEARNER
I20240503 02:15:13.539513 12737 consensus_queue.cc:260] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:13.544647 12824 tablet_copy_client.cc:308] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.197:45403
I20240503 02:15:13.545504 8972 tablet_copy_service.cc:140] P 99be4ff1114d4aaabcc537604fce54b3: Received BeginTabletCopySession request for tablet 95a8f2627c5c49a39291c96eb00fec18 from peer dabeea74472746538132a520710248ea ({username='slave'} at 127.23.104.202:55513)
I20240503 02:15:13.545727 8972 tablet_copy_service.cc:161] P 99be4ff1114d4aaabcc537604fce54b3: Beginning new tablet copy session on tablet 95a8f2627c5c49a39291c96eb00fec18 from peer dabeea74472746538132a520710248ea at {username='slave'} at 127.23.104.202:55513: session id = dabeea74472746538132a520710248ea-95a8f2627c5c49a39291c96eb00fec18
I20240503 02:15:13.548918 12737 ts_tablet_manager.cc:1398] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20240503 02:15:13.549832 8972 tablet_copy_source_session.cc:205] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:13.551506 12839 tablet_copy_client.cc:308] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:45923
I20240503 02:15:13.551630 9234 tablet_copy_service.cc:342] P cb90f34dc37b4b16b2a5b5a4ee012865: Request end of tablet copy session dabeea74472746538132a520710248ea-879167ff15db41a7b4ef21fa10c44b27 received from {username='slave'} at 127.23.104.202:35173
I20240503 02:15:13.551743 9234 tablet_copy_service.cc:433] P cb90f34dc37b4b16b2a5b5a4ee012865: ending tablet copy session dabeea74472746538132a520710248ea-879167ff15db41a7b4ef21fa10c44b27 on tablet 879167ff15db41a7b4ef21fa10c44b27 with peer dabeea74472746538132a520710248ea
I20240503 02:15:13.552606 8578 tablet_copy_service.cc:140] P 2a1893ab239f4163a06bdbf8ce579431: Received BeginTabletCopySession request for tablet 99e42dca2db54de5a9417df20242e2c2 from peer e512cb0d8f244ac6ab0e7150a0a8e2df ({username='slave'} at 127.23.104.200:41677)
I20240503 02:15:13.552794 8578 tablet_copy_service.cc:161] P 2a1893ab239f4163a06bdbf8ce579431: Beginning new tablet copy session on tablet 99e42dca2db54de5a9417df20242e2c2 from peer e512cb0d8f244ac6ab0e7150a0a8e2df at {username='slave'} at 127.23.104.200:41677: session id = e512cb0d8f244ac6ab0e7150a0a8e2df-99e42dca2db54de5a9417df20242e2c2
I20240503 02:15:13.552716 9213 consensus_queue.cc:237] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:13.555124 8578 tablet_copy_source_session.cc:205] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:13.557402 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a8f2627c5c49a39291c96eb00fec18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:13.558403 12839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e42dca2db54de5a9417df20242e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:13.563838 12824 tablet_copy_client.cc:783] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:13.564759 12824 tablet_copy_client.cc:647] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:13.565035 12839 tablet_copy_client.cc:783] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:13.565264 12839 tablet_copy_client.cc:647] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 1 WAL segments...
W20240503 02:15:13.567243 9149 consensus_peers.cc:485] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 -> Peer dabeea74472746538132a520710248ea (127.23.104.202:37759): Couldn't send request to peer dabeea74472746538132a520710248ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac90f5b4e0fc43e99b3f76fe92610153. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:13.567719 12824 tablet_copy_client.cc:523] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:13.568148 12839 tablet_copy_client.cc:523] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:13.568889 10815 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.569551 12703 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.570683 8689 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.571202 12533 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.575137 8557 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.575650 12533 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.578437 12824 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea: Bootstrap starting.
I20240503 02:15:13.579461 12839 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap starting.
I20240503 02:15:13.585791 8423 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.586827 9083 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.587476 12703 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.588048 12703 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.594883 8819 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.595485 12533 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.600813 12533 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.603961 8689 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.604163 8819 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.609680 8423 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.610765 8277 catalog_manager.cc:5484] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf reported cstate change: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.613801 8557 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.618422 10815 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.639593 10947 raft_consensus.cc:1203] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:15:13.644811 9083 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:13.657083 10815 raft_consensus.cc:1203] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:15:13.672456 12839 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:13.672914 12839 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap complete.
I20240503 02:15:13.673213 12839 ts_tablet_manager.cc:1367] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent bootstrapping tablet: real 0.098s user 0.059s sys 0.012s
I20240503 02:15:13.674050 12839 raft_consensus.cc:348] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:13.674384 12839 raft_consensus.cc:729] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e512cb0d8f244ac6ab0e7150a0a8e2df, State: Initialized, Role: LEARNER
I20240503 02:15:13.675096 12839 consensus_queue.cc:260] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:13.675882 12839 ts_tablet_manager.cc:1398] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240503 02:15:13.678668 8578 tablet_copy_service.cc:342] P 2a1893ab239f4163a06bdbf8ce579431: Request end of tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-99e42dca2db54de5a9417df20242e2c2 received from {username='slave'} at 127.23.104.200:41677
I20240503 02:15:13.678802 8578 tablet_copy_service.cc:433] P 2a1893ab239f4163a06bdbf8ce579431: ending tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-99e42dca2db54de5a9417df20242e2c2 on tablet 99e42dca2db54de5a9417df20242e2c2 with peer e512cb0d8f244ac6ab0e7150a0a8e2df
I20240503 02:15:13.678615 8819 consensus_queue.cc:237] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:13.680562 12824 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:13.680979 12824 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea: Bootstrap complete.
I20240503 02:15:13.681267 12824 ts_tablet_manager.cc:1367] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea: Time spent bootstrapping tablet: real 0.103s user 0.049s sys 0.004s
I20240503 02:15:13.681895 12824 raft_consensus.cc:348] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:13.682230 12824 raft_consensus.cc:729] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dabeea74472746538132a520710248ea, State: Initialized, Role: LEARNER
I20240503 02:15:13.682443 12824 consensus_queue.cc:260] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:13.683055 12824 ts_tablet_manager.cc:1398] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:15:13.684849 8972 tablet_copy_service.cc:342] P 99be4ff1114d4aaabcc537604fce54b3: Request end of tablet copy session dabeea74472746538132a520710248ea-95a8f2627c5c49a39291c96eb00fec18 received from {username='slave'} at 127.23.104.202:55513
I20240503 02:15:13.684944 8972 tablet_copy_service.cc:433] P 99be4ff1114d4aaabcc537604fce54b3: ending tablet copy session dabeea74472746538132a520710248ea-95a8f2627c5c49a39291c96eb00fec18 on tablet 95a8f2627c5c49a39291c96eb00fec18 with peer dabeea74472746538132a520710248ea
I20240503 02:15:13.697355 11079 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 02:15:13.697736 8755 consensus_peers.cc:485] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf -> Peer 4792082063bd4d58bc222c9c95b16335 (127.23.104.201:46877): Couldn't send request to peer 4792082063bd4d58bc222c9c95b16335. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b41db0e8b114c99ab1d8734ec8b008f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:13.698221 12812 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.701153 9083 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.701948 12812 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.702391 8952 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.703666 12812 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.708177 8689 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.708659 12812 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.711201 8557 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.711664 8423 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:13.711812 12690 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:13.712922 12812 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.713428 12812 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:13.714114 8689 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:13.717712 12585 raft_consensus.cc:1050] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d: attempting to promote NON_VOTER dabeea74472746538132a520710248ea to VOTER
I20240503 02:15:13.718452 8423 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:13.718878 12585 consensus_queue.cc:237] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:13.721825 8281 catalog_manager.cc:5484] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 reported cstate change: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "79e41f25066e4e5abbd71e48d545c6bf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:13.724254 12771 ts_tablet_manager.cc:902] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335: Initiating tablet copy from peer 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:15:13.736407 11079 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:13.744329 9083 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:13.744418 8952 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:13.755124 8557 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:13.775974 12771 tablet_copy_client.cc:308] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:45495
I20240503 02:15:13.786777 10815 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:13.786803 8557 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:13.787355 8689 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:13.788055 12648 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:13.788890 12632 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:13.791436 8710 tablet_copy_service.cc:140] P 846e89dd0b154d2b9bb7d4f315c9fc12: Received BeginTabletCopySession request for tablet fe35d536ab6348c58e90d680383bb763 from peer 4792082063bd4d58bc222c9c95b16335 ({username='slave'} at 127.23.104.201:42305)
I20240503 02:15:13.791582 8710 tablet_copy_service.cc:161] P 846e89dd0b154d2b9bb7d4f315c9fc12: Beginning new tablet copy session on tablet fe35d536ab6348c58e90d680383bb763 from peer 4792082063bd4d58bc222c9c95b16335 at {username='slave'} at 127.23.104.201:42305: session id = 4792082063bd4d58bc222c9c95b16335-fe35d536ab6348c58e90d680383bb763
I20240503 02:15:13.795358 8710 tablet_copy_source_session.cc:205] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:13.795573 11079 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:13.796368 12648 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:13.799841 8819 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:13.800496 8425 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:13.800634 12886 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:13.801599 9213 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:13.803898 12771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe35d536ab6348c58e90d680383bb763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:13.801151 12887 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.003s
I20240503 02:15:13.804247 12632 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:13.804934 12585 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:13.806535 12648 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:13.809413 11079 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:13.822783 8282 catalog_manager.cc:5484] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea reported cstate change: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:13.830276 12771 tablet_copy_client.cc:783] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:13.830544 12771 tablet_copy_client.cc:647] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:13.834547 12771 tablet_copy_client.cc:523] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:13.838822 10815 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:13.844635 8557 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:13.846849 9213 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:13.851681 12771 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335: Bootstrap starting.
I20240503 02:15:13.860316 8819 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:13.870357 11079 raft_consensus.cc:1203] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:15:13.884406 8425 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:13.899946 10947 raft_consensus.cc:1203] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:15:13.942242 10815 raft_consensus.cc:1203] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:15:13.957512 12771 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:13.957957 12771 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335: Bootstrap complete.
I20240503 02:15:13.958206 12771 ts_tablet_manager.cc:1367] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335: Time spent bootstrapping tablet: real 0.107s user 0.042s sys 0.000s
I20240503 02:15:13.958983 12771 raft_consensus.cc:348] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:13.959290 12771 raft_consensus.cc:729] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4792082063bd4d58bc222c9c95b16335, State: Initialized, Role: LEARNER
I20240503 02:15:13.959497 12771 consensus_queue.cc:260] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:13.960108 12771 ts_tablet_manager.cc:1398] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240503 02:15:13.960784 8710 tablet_copy_service.cc:342] P 846e89dd0b154d2b9bb7d4f315c9fc12: Request end of tablet copy session 4792082063bd4d58bc222c9c95b16335-fe35d536ab6348c58e90d680383bb763 received from {username='slave'} at 127.23.104.201:42305
I20240503 02:15:13.960920 8710 tablet_copy_service.cc:433] P 846e89dd0b154d2b9bb7d4f315c9fc12: ending tablet copy session 4792082063bd4d58bc222c9c95b16335-fe35d536ab6348c58e90d680383bb763 on tablet fe35d536ab6348c58e90d680383bb763 with peer 4792082063bd4d58bc222c9c95b16335
I20240503 02:15:13.964257 12824 ts_tablet_manager.cc:902] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea: Initiating tablet copy from peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:15:13.964771 12737 ts_tablet_manager.cc:902] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea: Initiating tablet copy from peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:15:13.981287 12824 tablet_copy_client.cc:308] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.199:32809
I20240503 02:15:13.982041 12737 tablet_copy_client.cc:308] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:41237
I20240503 02:15:13.982170 9234 tablet_copy_service.cc:140] P cb90f34dc37b4b16b2a5b5a4ee012865: Received BeginTabletCopySession request for tablet ac90f5b4e0fc43e99b3f76fe92610153 from peer dabeea74472746538132a520710248ea ({username='slave'} at 127.23.104.202:35173)
I20240503 02:15:13.982332 9234 tablet_copy_service.cc:161] P cb90f34dc37b4b16b2a5b5a4ee012865: Beginning new tablet copy session on tablet ac90f5b4e0fc43e99b3f76fe92610153 from peer dabeea74472746538132a520710248ea at {username='slave'} at 127.23.104.202:35173: session id = dabeea74472746538132a520710248ea-ac90f5b4e0fc43e99b3f76fe92610153
I20240503 02:15:13.982702 9103 tablet_copy_service.cc:140] P c399efcbba924863ac89d156dc07ac7d: Received BeginTabletCopySession request for tablet 1a30758cbd1d4845bb9eeb02e1a2942a from peer dabeea74472746538132a520710248ea ({username='slave'} at 127.23.104.202:46779)
I20240503 02:15:13.982867 9103 tablet_copy_service.cc:161] P c399efcbba924863ac89d156dc07ac7d: Beginning new tablet copy session on tablet 1a30758cbd1d4845bb9eeb02e1a2942a from peer dabeea74472746538132a520710248ea at {username='slave'} at 127.23.104.202:46779: session id = dabeea74472746538132a520710248ea-1a30758cbd1d4845bb9eeb02e1a2942a
I20240503 02:15:13.985342 9234 tablet_copy_source_session.cc:205] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:13.985857 9103 tablet_copy_source_session.cc:205] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:13.986722 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac90f5b4e0fc43e99b3f76fe92610153. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:13.987105 12737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a30758cbd1d4845bb9eeb02e1a2942a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:13.988240 10815 raft_consensus.cc:1203] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:15:13.995303 12824 tablet_copy_client.cc:783] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:13.995576 12824 tablet_copy_client.cc:647] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:13.997321 12737 tablet_copy_client.cc:783] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:13.997627 12737 tablet_copy_client.cc:647] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:13.997875 12824 tablet_copy_client.cc:523] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:13.999887 12737 tablet_copy_client.cc:523] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:14.004002 12737 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea: Bootstrap starting.
I20240503 02:15:14.005417 12824 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea: Bootstrap starting.
I20240503 02:15:14.031973 11078 raft_consensus.cc:1203] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:15:14.034816 12585 raft_consensus.cc:1050] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d: attempting to promote NON_VOTER 4792082063bd4d58bc222c9c95b16335 to VOTER
I20240503 02:15:14.036068 12585 consensus_queue.cc:237] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:14.044371 12687 raft_consensus.cc:1050] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3: attempting to promote NON_VOTER 4792082063bd4d58bc222c9c95b16335 to VOTER
I20240503 02:15:14.046099 12687 consensus_queue.cc:237] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:14.055505 10947 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.056257 9213 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.056564 8952 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.056788 10815 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.057358 12894 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.058084 12894 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.058786 12887 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.060217 12648 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.067714 8689 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.068394 12887 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.081477 8557 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.086042 12887 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.089501 8425 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.091629 12585 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.098065 12887 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.105659 9212 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.106460 8690 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.107031 12504 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.107005 10816 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.107465 9213 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.109236 9083 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.109817 8952 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.113497 12711 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.114032 12711 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.114171 10947 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.117671 8558 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.126238 8282 catalog_manager.cc:5484] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.131018 12711 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.130967 8425 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.133008 10946 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.134547 8688 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.150857 9081 consensus_queue.cc:237] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:14.154186 12824 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:14.154623 12824 tablet_bootstrap.cc:492] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea: Bootstrap complete.
I20240503 02:15:14.154881 12824 ts_tablet_manager.cc:1367] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea: Time spent bootstrapping tablet: real 0.150s user 0.053s sys 0.012s
I20240503 02:15:14.156409 9213 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.157156 12887 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.158345 8423 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.158856 10945 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.159009 12504 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.160534 8557 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.161115 12887 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.161865 12887 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.165908 8688 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.166093 8952 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.166482 12887 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.167050 12648 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.167378 12824 raft_consensus.cc:348] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:14.167758 12824 raft_consensus.cc:729] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dabeea74472746538132a520710248ea, State: Initialized, Role: LEARNER
I20240503 02:15:14.168859 12824 consensus_queue.cc:260] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:14.169188 10816 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.169808 12648 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.170068 10815 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.170660 12504 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.171131 12824 ts_tablet_manager.cc:1398] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240503 02:15:14.172010 9234 tablet_copy_service.cc:342] P cb90f34dc37b4b16b2a5b5a4ee012865: Request end of tablet copy session dabeea74472746538132a520710248ea-ac90f5b4e0fc43e99b3f76fe92610153 received from {username='slave'} at 127.23.104.202:35173
I20240503 02:15:14.172163 9234 tablet_copy_service.cc:433] P cb90f34dc37b4b16b2a5b5a4ee012865: ending tablet copy session dabeea74472746538132a520710248ea-ac90f5b4e0fc43e99b3f76fe92610153 on tablet ac90f5b4e0fc43e99b3f76fe92610153 with peer dabeea74472746538132a520710248ea
I20240503 02:15:14.181615 12648 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.186203 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5028c92b346845c38b3356a0d8c7507f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:14.187661 9213 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.189114 10945 raft_consensus.cc:1203] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:15:14.194887 8557 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.195804 8282 catalog_manager.cc:5484] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.199401 10947 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.202421 12711 raft_consensus.cc:1050] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3: attempting to promote NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df to VOTER
I20240503 02:15:14.203616 12711 consensus_queue.cc:237] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:14.208107 8403 tablet_service.cc:1514] Processing DeleteTablet for tablet 5028c92b346845c38b3356a0d8c7507f with delete_type TABLET_DATA_TOMBSTONED (TS 4c668b2e2d5c4edba46cbc5c36065c0c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41418
I20240503 02:15:14.208971 12969 tablet_replica.cc:330] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c: stopping tablet replica
I20240503 02:15:14.209945 12969 raft_consensus.cc:2229] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:14.210448 12969 raft_consensus.cc:2258] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:14.212455 10947 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.213461 9081 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.214229 12504 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.214529 12969 ts_tablet_manager.cc:1875] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:14.215008 12504 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.217773 9213 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.218538 12504 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.224145 12969 ts_tablet_manager.cc:1888] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:15:14.224351 10814 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.224387 12969 log.cc:1196] T 5028c92b346845c38b3356a0d8c7507f P 4c668b2e2d5c4edba46cbc5c36065c0c: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/5028c92b346845c38b3356a0d8c7507f
I20240503 02:15:14.224689 8687 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.225543 8267 catalog_manager.cc:4830] TS 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635): tablet 5028c92b346845c38b3356a0d8c7507f (table rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]) successfully deleted
I20240503 02:15:14.228533 8557 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.229468 12504 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.229840 12976 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.230692 12976 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.237421 12771 ts_tablet_manager.cc:902] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335: Initiating tablet copy from peer 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747)
I20240503 02:15:14.240211 10814 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.239730 12504 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.247026 8279 catalog_manager.cc:5484] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df reported cstate change: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.253249 8687 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.268990 12737 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:14.269557 12737 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea: Bootstrap complete.
I20240503 02:15:14.269953 12737 ts_tablet_manager.cc:1367] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea: Time spent bootstrapping tablet: real 0.266s user 0.049s sys 0.010s
I20240503 02:15:14.271019 12737 raft_consensus.cc:348] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:14.271528 12737 raft_consensus.cc:729] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dabeea74472746538132a520710248ea, State: Initialized, Role: LEARNER
I20240503 02:15:14.274825 12737 consensus_queue.cc:260] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:14.276441 12737 ts_tablet_manager.cc:1398] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240503 02:15:14.277701 9103 tablet_copy_service.cc:342] P c399efcbba924863ac89d156dc07ac7d: Request end of tablet copy session dabeea74472746538132a520710248ea-1a30758cbd1d4845bb9eeb02e1a2942a received from {username='slave'} at 127.23.104.202:46779
I20240503 02:15:14.277881 9103 tablet_copy_service.cc:433] P c399efcbba924863ac89d156dc07ac7d: ending tablet copy session dabeea74472746538132a520710248ea-1a30758cbd1d4845bb9eeb02e1a2942a on tablet 1a30758cbd1d4845bb9eeb02e1a2942a with peer dabeea74472746538132a520710248ea
I20240503 02:15:14.277523 10947 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.283155 9081 consensus_queue.cc:237] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:14.292104 10814 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.292455 8424 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.293417 12648 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.294126 12648 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.297480 8686 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.187657 8688 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.300403 9212 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.301916 11078 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.302345 12632 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.303155 12648 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.304279 12886 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.314148 12887 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.323513 11078 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.332121 8952 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.332906 10816 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.337006 9212 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.354023 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df68cc6032b14cae9e1cac074976baf8 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:14.355414 10814 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.362687 8281 catalog_manager.cc:5484] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df reported cstate change: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.367952 8537 tablet_service.cc:1514] Processing DeleteTablet for tablet df68cc6032b14cae9e1cac074976baf8 with delete_type TABLET_DATA_TOMBSTONED (TS 2a1893ab239f4163a06bdbf8ce579431 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51278
I20240503 02:15:14.382007 12987 tablet_replica.cc:330] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: stopping tablet replica
I20240503 02:15:14.382658 12987 raft_consensus.cc:2229] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:14.383570 12987 raft_consensus.cc:2258] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:14.390903 12987 ts_tablet_manager.cc:1875] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:15:14.397594 9017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.195:45495, user_credentials={real_user=slave}} blocked reactor thread for 95817.7us
I20240503 02:15:14.397711 9213 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.402928 8819 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.403759 12987 ts_tablet_manager.cc:1888] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:15:14.403136 8424 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.404116 12987 log.cc:1196] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/df68cc6032b14cae9e1cac074976baf8
I20240503 02:15:14.405473 8269 catalog_manager.cc:4830] TS 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): tablet df68cc6032b14cae9e1cac074976baf8 (table rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]) successfully deleted
I20240503 02:15:14.411087 12976 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.417032 8424 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.427109 11078 raft_consensus.cc:1203] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:15:14.427606 8423 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.428227 12975 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.430413 8423 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.438176 12711 raft_consensus.cc:1050] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3: attempting to promote NON_VOTER 4792082063bd4d58bc222c9c95b16335 to VOTER
I20240503 02:15:14.445717 12860 raft_consensus.cc:1050] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865: attempting to promote NON_VOTER dabeea74472746538132a520710248ea to VOTER
I20240503 02:15:14.447366 12860 consensus_queue.cc:237] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:14.453289 8557 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.463006 8279 catalog_manager.cc:5484] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.464552 10816 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.470387 11078 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.471148 12862 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.477013 10816 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.478055 12862 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.478489 12687 raft_consensus.cc:1050] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3: attempting to promote NON_VOTER dabeea74472746538132a520710248ea to VOTER
I20240503 02:15:14.480206 12687 consensus_queue.cc:237] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:14.483510 8689 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.484241 8952 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.484947 8557 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.486112 12862 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.487864 11079 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.488492 9213 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.488440 12861 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.488747 12976 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.489120 12504 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.489228 12860 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.492959 8558 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.496214 8425 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.502425 12993 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.498250 12860 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.012s
I20240503 02:15:14.527104 9081 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.533047 12862 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.533161 11077 raft_consensus.cc:1203] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:15:14.536470 12632 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.454026 9214 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.539079 8557 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.539731 12687 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.542068 12585 raft_consensus.cc:1050] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d: attempting to promote NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df to VOTER
I20240503 02:15:14.543431 12585 consensus_queue.cc:237] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:14.568383 12859 raft_consensus.cc:1050] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431: attempting to promote NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df to VOTER
I20240503 02:15:14.570151 12859 consensus_queue.cc:237] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:14.576053 12861 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.576851 8949 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.585105 12899 raft_consensus.cc:1050] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12: attempting to promote NON_VOTER 4792082063bd4d58bc222c9c95b16335 to VOTER
I20240503 02:15:14.572085 8557 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.586385 12899 consensus_queue.cc:237] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:14.586185 9081 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.586733 10816 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.591710 8689 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.594864 10945 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.595535 12900 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.597508 8279 catalog_manager.cc:5484] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:14.600394 11079 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.600435 8819 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.601225 12900 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.601516 8952 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.602222 12900 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.602823 9213 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.603710 9081 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.604300 11953 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.604799 11953 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.615923 8557 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.616535 12899 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.618942 8424 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.621009 12899 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.588436 8558 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.632855 11079 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.639858 10814 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.640588 12886 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.646411 9212 consensus_queue.cc:237] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:14.651600 8689 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.652997 8952 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.653234 9083 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.654109 12862 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.654605 12862 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.655033 12862 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.654606 12900 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.660579 11077 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.661978 11079 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.662082 8689 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.663383 12861 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.668264 8556 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.669020 8555 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.668407 8279 catalog_manager.cc:5484] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea reported cstate change: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.669663 12860 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.674854 8952 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.676277 12886 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.690218 8689 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.690228 8423 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.700264 8269 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 879167ff15db41a7b4ef21fa10c44b27 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:14.729737 8281 catalog_manager.cc:5484] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 reported cstate change: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.734867 8690 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.734895 10947 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.736424 8403 tablet_service.cc:1514] Processing DeleteTablet for tablet 879167ff15db41a7b4ef21fa10c44b27 with delete_type TABLET_DATA_TOMBSTONED (TS 4c668b2e2d5c4edba46cbc5c36065c0c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41418
I20240503 02:15:14.739187 9211 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.739763 12632 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.744864 11076 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.754407 12632 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.756214 8422 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.757790 8819 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.759266 12632 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:15:14.765494 8886 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.201:46877, user_credentials={real_user=slave}} blocked reactor thread for 334746us
I20240503 02:15:14.766400 10813 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.766287 9083 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.767473 8686 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.771216 10946 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.772773 8685 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.773867 9082 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.774669 9080 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.776145 10947 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.777853 12765 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.782159 8687 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.785001 8422 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.787369 12993 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.788460 12990 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.789021 12687 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.789587 12993 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.791891 13021 tablet_replica.cc:330] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c: stopping tablet replica
I20240503 02:15:14.792219 13021 raft_consensus.cc:2229] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:14.792508 13021 pending_rounds.cc:62] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c: Trying to abort 1 pending ops.
I20240503 02:15:14.792698 13021 pending_rounds.cc:68] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c: Aborting op as it isn't in flight: id { term: 1 index: 9 } timestamp: 7023421499171991552 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "879167ff15db41a7b4ef21fa10c44b27" old_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } } new_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } } }
I20240503 02:15:14.793624 13021 raft_consensus.cc:2894] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Aborting config change with OpId 1.9: Aborted: Op aborted
I20240503 02:15:14.793841 13021 raft_consensus.cc:2866] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:15:14.794049 13021 raft_consensus.cc:2258] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:14.800047 13021 ts_tablet_manager.cc:1875] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:14.800942 9210 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.802445 12993 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.808436 9083 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.810220 8556 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.812422 11076 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.818203 8279 catalog_manager.cc:5484] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.820583 9210 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.822584 12632 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.824955 12632 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.835776 8689 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.846740 13021 ts_tablet_manager.cc:1888] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:15:14.847389 13021 log.cc:1196] T 879167ff15db41a7b4ef21fa10c44b27 P 4c668b2e2d5c4edba46cbc5c36065c0c: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/879167ff15db41a7b4ef21fa10c44b27
I20240503 02:15:14.848336 9082 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.848574 8952 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.851708 8689 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.852262 11076 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.852387 8424 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.852347 12765 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.852756 8267 catalog_manager.cc:4830] TS 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635): tablet 879167ff15db41a7b4ef21fa10c44b27 (table rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]) successfully deleted
I20240503 02:15:14.852998 12765 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.856010 9083 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.857537 12765 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.858356 12765 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.865132 10947 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.869035 9212 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.869562 12765 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.875336 8952 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:14.876442 12765 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.877393 12585 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.867344 8423 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.797381 12711 consensus_queue.cc:237] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:14.887102 8281 catalog_manager.cc:5484] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d reported cstate change: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:15:14.889715 8690 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.890121 10814 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.894008 9212 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.894255 8422 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.899605 8821 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.902698 10946 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.908694 9083 consensus_queue.cc:237] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:14.912478 8821 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.913056 12585 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.915396 10943 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.917382 8557 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.918138 12860 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.922328 12859 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.928185 9082 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:14.930967 12771 tablet_copy_client.cc:308] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.196:34747
I20240503 02:15:14.931782 10815 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.932627 8841 tablet_copy_service.cc:140] P 79e41f25066e4e5abbd71e48d545c6bf: Received BeginTabletCopySession request for tablet 9b41db0e8b114c99ab1d8734ec8b008f from peer 4792082063bd4d58bc222c9c95b16335 ({username='slave'} at 127.23.104.201:57903)
I20240503 02:15:14.932790 8841 tablet_copy_service.cc:161] P 79e41f25066e4e5abbd71e48d545c6bf: Beginning new tablet copy session on tablet 9b41db0e8b114c99ab1d8734ec8b008f from peer 4792082063bd4d58bc222c9c95b16335 at {username='slave'} at 127.23.104.201:57903: session id = 4792082063bd4d58bc222c9c95b16335-9b41db0e8b114c99ab1d8734ec8b008f
I20240503 02:15:14.932718 12860 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.934490 10815 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.935571 8841 tablet_copy_source_session.cc:205] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:14.942916 10813 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.947163 8557 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:14.954535 9210 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.955317 12986 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.956332 8556 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.957597 12986 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:14.958858 8558 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.959767 12771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b41db0e8b114c99ab1d8734ec8b008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:14.959390 9081 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.963728 11076 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.966976 10812 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.971122 8425 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.971354 12648 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.972313 12886 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:14.971550 8277 catalog_manager.cc:5484] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea reported cstate change: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.974750 12886 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.977578 8690 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.980063 8425 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.981704 8688 consensus_queue.cc:237] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:14.981845 9214 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.982385 8423 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.983884 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b71357f9f8ef449189f485b3246869f1 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:14.984889 10814 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.987730 8821 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:14.989831 8273 catalog_manager.cc:5484] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d reported cstate change: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:15:14.995630 8670 tablet_service.cc:1514] Processing DeleteTablet for tablet b71357f9f8ef449189f485b3246869f1 with delete_type TABLET_DATA_TOMBSTONED (TS 846e89dd0b154d2b9bb7d4f315c9fc12 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36802
I20240503 02:15:14.997524 10944 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:14.998924 12632 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.001456 12862 raft_consensus.cc:1050] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865: attempting to promote NON_VOTER dabeea74472746538132a520710248ea to VOTER
I20240503 02:15:15.002758 12862 consensus_queue.cc:237] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:15.004722 10944 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.004233 9213 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.005401 11953 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.010196 12585 raft_consensus.cc:1050] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d: attempting to promote NON_VOTER dabeea74472746538132a520710248ea to VOTER
I20240503 02:15:15.011255 11076 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.011895 11953 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.011698 12585 consensus_queue.cc:237] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:15.014235 10946 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.017149 8952 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.021467 10944 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.021683 11078 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.022228 12993 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.022770 12993 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.025946 9082 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.026461 12990 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.026551 8689 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.029022 12993 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.030102 8556 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.030718 12993 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.031260 8952 consensus_queue.cc:237] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:15.034587 9213 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.035285 12993 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.036983 10812 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.037715 12975 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.042917 8555 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.044304 9213 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.044884 12990 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.044989 8557 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.045758 8422 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.045812 10944 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.046290 12975 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.046785 12975 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.047937 12975 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.049772 9081 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.049969 12687 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.051210 12962 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.053085 9083 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.054247 8268 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7a434e2df0a4ea0b2ab5c20034033c6 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:15.056340 10812 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.058758 8952 consensus_queue.cc:237] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:15.056521 9214 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.062583 8282 catalog_manager.cc:5484] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d reported cstate change: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.065971 8555 consensus_queue.cc:237] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:15.067950 10944 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.068046 8424 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.066882 8558 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.082399 10812 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.082443 10945 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.083034 12990 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.083223 9213 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.083853 12993 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.084380 12993 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.084488 10945 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.085122 12886 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.087100 11078 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea [term 1 LEARNER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.087131 9212 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.087586 12886 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.086351 8422 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.087883 12632 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.088706 12711 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.089246 8685 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.089989 8821 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.089951 12632 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.091872 12632 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.091823 8952 consensus_queue.cc:237] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:15.092900 12993 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.096827 8421 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.098431 8557 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.099046 12632 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.099864 8268 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:15.102710 10943 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.102648 10812 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.103528 8279 catalog_manager.cc:5484] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.109025 8537 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 with delete_type TABLET_DATA_TOMBSTONED (TS 2a1893ab239f4163a06bdbf8ce579431 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51278
I20240503 02:15:15.110966 10942 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.112147 9210 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.112949 12976 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.114180 12990 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.116796 11078 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.117430 12711 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.120196 8553 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.121090 9083 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.121687 8687 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.122503 12990 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.122920 9081 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.120591 9213 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.123435 12976 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.123922 12976 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.124365 12976 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.125427 12976 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.125779 8553 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.126711 13065 tablet_replica.cc:330] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431: stopping tablet replica
I20240503 02:15:15.126910 13065 raft_consensus.cc:2229] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:15.127184 13065 raft_consensus.cc:2258] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:15.128788 11078 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.131927 8268 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 95a8f2627c5c49a39291c96eb00fec18 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:15.133396 10945 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.134250 9210 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.135499 8279 catalog_manager.cc:5484] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.137030 9083 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.137509 8687 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.143878 8423 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.144456 12976 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.144691 8422 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.145915 13065 ts_tablet_manager.cc:1875] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:15.145474 9082 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.146984 12585 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.149905 8669 tablet_service.cc:1514] Processing DeleteTablet for tablet a7a434e2df0a4ea0b2ab5c20034033c6 with delete_type TABLET_DATA_TOMBSTONED (TS 846e89dd0b154d2b9bb7d4f315c9fc12 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36802
I20240503 02:15:15.155881 13044 tablet_replica.cc:330] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12: stopping tablet replica
I20240503 02:15:15.156392 13044 raft_consensus.cc:2229] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:15.156782 13044 raft_consensus.cc:2258] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:15.156886 10815 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.157917 13033 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.159977 8403 tablet_service.cc:1514] Processing DeleteTablet for tablet 95a8f2627c5c49a39291c96eb00fec18 with delete_type TABLET_DATA_TOMBSTONED (TS 4c668b2e2d5c4edba46cbc5c36065c0c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41418
I20240503 02:15:15.160418 13021 tablet_replica.cc:330] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: stopping tablet replica
I20240503 02:15:15.160596 13021 raft_consensus.cc:2229] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:15.160878 13021 raft_consensus.cc:2258] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:15.160791 12976 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.163689 13021 ts_tablet_manager.cc:1875] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:15.163852 8553 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.168449 8690 raft_consensus.cc:1261] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.168404 9210 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.175220 8952 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.175729 12993 consensus_queue.cc:1035] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.175894 11953 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.178741 11953 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.179637 8685 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.182248 8689 raft_consensus.cc:2932] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.184619 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe35d536ab6348c58e90d680383bb763 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:15.186564 9083 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.190778 9213 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.191401 12900 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.191118 8952 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.193423 9214 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.201647 8282 catalog_manager.cc:5484] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.204649 13021 ts_tablet_manager.cc:1888] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:15:15.204784 13021 log.cc:1196] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/95a8f2627c5c49a39291c96eb00fec18
I20240503 02:15:15.205049 13044 ts_tablet_manager.cc:1875] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:15.205657 8267 catalog_manager.cc:4830] TS 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635): tablet 95a8f2627c5c49a39291c96eb00fec18 (table rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]) successfully deleted
I20240503 02:15:15.206823 11077 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea [term 1 LEARNER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.206233 12894 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.207427 12860 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.211118 8421 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.211740 12900 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.213647 10942 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.214627 8421 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.215953 11079 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.218067 9081 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.217582 8821 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.217582 10941 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.218917 8273 catalog_manager.cc:5484] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:15:15.220111 12899 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.221101 8557 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.223080 9210 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.224057 8279 catalog_manager.cc:5484] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf reported cstate change: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.225389 11079 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.226182 8424 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.202143 10947 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.234473 11076 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.234815 9081 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.239125 8425 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.239706 11075 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.239931 13033 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.240535 13033 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.242188 9082 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.242779 13033 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.244963 8952 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.246345 13033 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.250249 8688 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.251003 10814 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.252143 9082 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.253676 12862 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.253918 8819 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.254523 12860 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.254879 9079 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.257180 12765 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.258657 12860 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.256937 8685 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.259554 13018 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.260836 8952 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.265846 8425 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.266841 11077 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.268496 8423 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.271807 12862 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.288445 8282 catalog_manager.cc:5484] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:15:15.297142 13065 ts_tablet_manager.cc:1888] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:15:15.297886 13065 log.cc:1196] T 3c28f0e30cfe46fda64dd2b0fd7847c6 P 2a1893ab239f4163a06bdbf8ce579431: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/3c28f0e30cfe46fda64dd2b0fd7847c6
I20240503 02:15:15.302543 12771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.23.104.196:43139 (request call id 21) took 1065 ms. Trace:
I20240503 02:15:15.306380 13018 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.306560 8423 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.309002 9082 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.311681 8421 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.312860 10814 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.312873 8819 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.315711 8670 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e42dca2db54de5a9417df20242e2c2 with delete_type TABLET_DATA_TOMBSTONED (TS 846e89dd0b154d2b9bb7d4f315c9fc12 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36802
I20240503 02:15:15.302805 12771 rpcz_store.cc:276] 0503 02:15:14.237025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:15:14.237114 (+ 89us) service_pool.cc:225] Handling call
0503 02:15:14.237498 (+ 384us) ts_tablet_manager.cc:903] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335: Initiating tablet copy from peer 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747)
0503 02:15:14.959231 (+721733us) tablet_copy_client.cc:353] Tablet copy session begun
0503 02:15:15.013847 (+ 54616us) tablet_metadata.cc:641] Metadata flushed
0503 02:15:15.013960 (+ 113us) tablet_copy_client.cc:482] Wrote new tablet metadata
0503 02:15:15.302265 (+288305us) tablet_copy_client.cc:403] Wrote new consensus metadata
0503 02:15:15.302376 (+ 111us) tablet_replica.cc:205] Creating consensus instance
0503 02:15:15.302483 (+ 107us) ts_tablet_manager.cc:958] Replica registered.
0503 02:15:15.302506 (+ 23us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":84,"tablet-copy.queue_time_us":47,"thread_start_us":693054,"threads_started":4}
I20240503 02:15:15.316609 12771 tablet_copy_client.cc:783] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:15.316880 12771 tablet_copy_client.cc:647] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:15.319337 8282 catalog_manager.cc:5484] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:15:15.320339 11077 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.328078 9210 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.328982 8269 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99e42dca2db54de5a9417df20242e2c2 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:15.330682 8952 consensus_queue.cc:237] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:15.329998 10814 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.335412 8269 catalog_manager.cc:4830] TS 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): tablet 3c28f0e30cfe46fda64dd2b0fd7847c6 (table rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]) successfully deleted
I20240503 02:15:15.329906 8688 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.336776 9210 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.337471 12711 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.338032 8537 tablet_service.cc:1514] Processing DeleteTablet for tablet fe35d536ab6348c58e90d680383bb763 with delete_type TABLET_DATA_TOMBSTONED (TS 2a1893ab239f4163a06bdbf8ce579431 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51278
I20240503 02:15:15.338526 8555 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:15.339059 13018 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:15.339572 12765 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.341666 13065 tablet_replica.cc:330] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431: stopping tablet replica
I20240503 02:15:15.342048 13065 raft_consensus.cc:2229] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:15.342347 13065 raft_consensus.cc:2258] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:15.342281 9210 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:15.343118 9081 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.343693 12990 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.347295 13065 ts_tablet_manager.cc:1875] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:15.351495 10947 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.353186 12976 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.355610 8685 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.356213 12976 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.359515 12771 tablet_copy_client.cc:523] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:15.359499 12711 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.360325 8422 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.364424 8268 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 64c52f9977ae4eb4ac803be99b6426c2 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:15.365044 8553 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.369643 9210 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.371078 8685 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.371840 8282 catalog_manager.cc:5484] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:15:15.375612 13044 ts_tablet_manager.cc:1888] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:15:15.376374 13044 log.cc:1196] T b71357f9f8ef449189f485b3246869f1 P 846e89dd0b154d2b9bb7d4f315c9fc12: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/b71357f9f8ef449189f485b3246869f1
I20240503 02:15:15.376576 11076 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.376910 8537 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c52f9977ae4eb4ac803be99b6426c2 with delete_type TABLET_DATA_TOMBSTONED (TS 2a1893ab239f4163a06bdbf8ce579431 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51278
I20240503 02:15:15.378316 12711 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.379951 8267 catalog_manager.cc:4830] TS 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): tablet b71357f9f8ef449189f485b3246869f1 (table rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]) successfully deleted
I20240503 02:15:15.380005 12993 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.380998 13044 tablet_replica.cc:330] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12: stopping tablet replica
I20240503 02:15:15.380460 11077 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.382081 8421 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.386749 13065 ts_tablet_manager.cc:1888] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:15:15.390333 13044 raft_consensus.cc:2229] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:15.390842 13044 raft_consensus.cc:2258] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:15.391829 13065 log.cc:1196] T fe35d536ab6348c58e90d680383bb763 P 2a1893ab239f4163a06bdbf8ce579431: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/fe35d536ab6348c58e90d680383bb763
I20240503 02:15:15.361114 9083 consensus_queue.cc:237] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:15.394233 13044 ts_tablet_manager.cc:1875] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:15.373675 9082 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.401276 10947 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:15.404951 13065 tablet_replica.cc:330] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431: stopping tablet replica
I20240503 02:15:15.404964 10946 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.404964 8269 catalog_manager.cc:4830] TS 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): tablet fe35d536ab6348c58e90d680383bb763 (table rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]) successfully deleted
I20240503 02:15:15.405162 13065 raft_consensus.cc:2229] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:15.405694 13065 raft_consensus.cc:2258] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:15.408382 13065 ts_tablet_manager.cc:1875] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:15.412014 12585 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.413753 12771 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335: Bootstrap starting.
I20240503 02:15:15.419770 11077 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.420634 9210 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.421468 12894 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.424448 12894 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.428478 12894 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.432744 11077 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.442711 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1a30758cbd1d4845bb9eeb02e1a2942a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:15.445214 8282 catalog_manager.cc:5484] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea reported cstate change: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.447808 8819 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.448647 12585 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.451231 8819 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.457363 8403 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a30758cbd1d4845bb9eeb02e1a2942a with delete_type TABLET_DATA_TOMBSTONED (TS 4c668b2e2d5c4edba46cbc5c36065c0c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41418
I20240503 02:15:15.457728 13021 tablet_replica.cc:330] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c: stopping tablet replica
I20240503 02:15:15.457969 13021 raft_consensus.cc:2229] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:15.458374 13021 raft_consensus.cc:2258] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:15.459542 8688 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.459681 13021 ts_tablet_manager.cc:1875] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:15.460239 12585 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.461942 8688 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.467017 13021 ts_tablet_manager.cc:1888] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:15:15.467007 13044 ts_tablet_manager.cc:1888] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:15:15.467208 13021 log.cc:1196] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4c668b2e2d5c4edba46cbc5c36065c0c: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/1a30758cbd1d4845bb9eeb02e1a2942a
I20240503 02:15:15.468062 8267 catalog_manager.cc:4830] TS 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635): tablet 1a30758cbd1d4845bb9eeb02e1a2942a (table rebalance_test_table_6 [id=29a819c57e47416886eb3dc5d96dd3c3]) successfully deleted
I20240503 02:15:15.467154 13044 log.cc:1196] T a7a434e2df0a4ea0b2ab5c20034033c6 P 846e89dd0b154d2b9bb7d4f315c9fc12: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/a7a434e2df0a4ea0b2ab5c20034033c6
I20240503 02:15:15.472210 8553 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.472908 12632 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.475164 8267 catalog_manager.cc:4830] TS 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): tablet a7a434e2df0a4ea0b2ab5c20034033c6 (table rebalance_test_table_4 [id=d237050aaedb40caaac24e9bd5d0352e]) successfully deleted
I20240503 02:15:15.475224 8553 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.483955 10947 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.484503 9210 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.493376 13065 ts_tablet_manager.cc:1888] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:15:15.493786 13065 log.cc:1196] T 64c52f9977ae4eb4ac803be99b6426c2 P 2a1893ab239f4163a06bdbf8ce579431: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/64c52f9977ae4eb4ac803be99b6426c2
I20240503 02:15:15.495751 8269 catalog_manager.cc:4830] TS 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): tablet 64c52f9977ae4eb4ac803be99b6426c2 (table rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]) successfully deleted
I20240503 02:15:15.516126 13044 tablet_replica.cc:330] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: stopping tablet replica
I20240503 02:15:15.516418 13044 raft_consensus.cc:2229] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:15.516798 13044 raft_consensus.cc:2258] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:15.517814 13044 ts_tablet_manager.cc:1875] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:15.527663 13044 ts_tablet_manager.cc:1888] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:15:15.527854 13044 log.cc:1196] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/99e42dca2db54de5a9417df20242e2c2
I20240503 02:15:15.528584 8267 catalog_manager.cc:4830] TS 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): tablet 99e42dca2db54de5a9417df20242e2c2 (table rebalance_test_table_1 [id=c56525a2930b410fab4415b138497987]) successfully deleted
I20240503 02:15:15.624030 9210 consensus_queue.cc:237] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
W20240503 02:15:15.667445 9149 consensus_peers.cc:485] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 -> Peer dabeea74472746538132a520710248ea (127.23.104.202:37759): Couldn't send request to peer dabeea74472746538132a520710248ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00fc93702514499eb32902238c46356d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:15.668615 10947 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:15.668784 8819 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:15.669495 9083 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:15.670799 8688 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:15.670822 12860 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:15.671023 8421 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:15.671387 12860 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:15.671770 12860 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:15.672485 8553 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:15.673028 12861 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:15.680742 12861 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:15.681418 12861 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:15.705520 13018 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:15.707741 10946 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:15.710752 12771 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:15.711340 12771 tablet_bootstrap.cc:492] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335: Bootstrap complete.
I20240503 02:15:15.711665 12771 ts_tablet_manager.cc:1367] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335: Time spent bootstrapping tablet: real 0.298s user 0.059s sys 0.007s
I20240503 02:15:15.713013 8819 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:15.714752 12771 raft_consensus.cc:348] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:15.714974 8278 catalog_manager.cc:5484] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 reported cstate change: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:15.715598 12771 raft_consensus.cc:729] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4792082063bd4d58bc222c9c95b16335, State: Initialized, Role: LEARNER
I20240503 02:15:15.715858 12771 consensus_queue.cc:260] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:15.716626 12771 ts_tablet_manager.cc:1398] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:15:15.717808 8841 tablet_copy_service.cc:342] P 79e41f25066e4e5abbd71e48d545c6bf: Request end of tablet copy session 4792082063bd4d58bc222c9c95b16335-9b41db0e8b114c99ab1d8734ec8b008f received from {username='slave'} at 127.23.104.201:57903
I20240503 02:15:15.717965 8841 tablet_copy_service.cc:433] P 79e41f25066e4e5abbd71e48d545c6bf: ending tablet copy session 4792082063bd4d58bc222c9c95b16335-9b41db0e8b114c99ab1d8734ec8b008f on tablet 9b41db0e8b114c99ab1d8734ec8b008f with peer 4792082063bd4d58bc222c9c95b16335
I20240503 02:15:15.720472 8421 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:15.723219 8685 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:15.730098 9083 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:15.749995 8555 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:15.886783 9210 consensus_queue.cc:237] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:15.889711 11077 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.890708 12860 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.892246 9083 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.892298 8421 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.894203 13018 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.894665 13018 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.897115 8553 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.898223 12862 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.898386 10812 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.900144 12861 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.901242 12861 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.902752 8553 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.905572 8421 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.907495 10812 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.908638 11077 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.909804 8821 raft_consensus.cc:1261] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:15.910368 8269 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac90f5b4e0fc43e99b3f76fe92610153 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:15.911271 12862 consensus_queue.cc:1035] T ac90f5b4e0fc43e99b3f76fe92610153 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:15.912559 8282 catalog_manager.cc:5484] T ac90f5b4e0fc43e99b3f76fe92610153 P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.912832 8821 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.919461 8670 tablet_service.cc:1514] Processing DeleteTablet for tablet ac90f5b4e0fc43e99b3f76fe92610153 with delete_type TABLET_DATA_TOMBSTONED (TS 846e89dd0b154d2b9bb7d4f315c9fc12 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36802
I20240503 02:15:15.919973 13044 tablet_replica.cc:330] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12: stopping tablet replica
I20240503 02:15:15.920172 13044 raft_consensus.cc:2229] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:15.920444 13044 raft_consensus.cc:2258] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:15.921547 9083 raft_consensus.cc:2932] T ac90f5b4e0fc43e99b3f76fe92610153 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:15.922821 13044 ts_tablet_manager.cc:1875] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:15.930469 13044 ts_tablet_manager.cc:1888] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:15:15.930666 13044 log.cc:1196] T ac90f5b4e0fc43e99b3f76fe92610153 P 846e89dd0b154d2b9bb7d4f315c9fc12: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/ac90f5b4e0fc43e99b3f76fe92610153
I20240503 02:15:15.932504 8267 catalog_manager.cc:4830] TS 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): tablet ac90f5b4e0fc43e99b3f76fe92610153 (table rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]) successfully deleted
I20240503 02:15:16.065470 10946 raft_consensus.cc:1203] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:15:16.070665 13189 ts_tablet_manager.cc:902] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea: Initiating tablet copy from peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:15:16.071848 13189 tablet_copy_client.cc:308] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.199:32809
I20240503 02:15:16.072665 9234 tablet_copy_service.cc:140] P cb90f34dc37b4b16b2a5b5a4ee012865: Received BeginTabletCopySession request for tablet 00fc93702514499eb32902238c46356d from peer dabeea74472746538132a520710248ea ({username='slave'} at 127.23.104.202:35173)
I20240503 02:15:16.072834 9234 tablet_copy_service.cc:161] P cb90f34dc37b4b16b2a5b5a4ee012865: Beginning new tablet copy session on tablet 00fc93702514499eb32902238c46356d from peer dabeea74472746538132a520710248ea at {username='slave'} at 127.23.104.202:35173: session id = dabeea74472746538132a520710248ea-00fc93702514499eb32902238c46356d
I20240503 02:15:16.074982 9234 tablet_copy_source_session.cc:205] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:16.076185 13189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fc93702514499eb32902238c46356d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:16.081377 13189 tablet_copy_client.cc:783] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:16.081637 13189 tablet_copy_client.cc:647] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:16.083122 13189 tablet_copy_client.cc:523] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:16.085253 13189 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea: Bootstrap starting.
I20240503 02:15:16.117198 13189 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:16.117640 13189 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea: Bootstrap complete.
I20240503 02:15:16.117883 13189 ts_tablet_manager.cc:1367] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea: Time spent bootstrapping tablet: real 0.033s user 0.031s sys 0.000s
I20240503 02:15:16.118472 13189 raft_consensus.cc:348] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:16.118685 13189 raft_consensus.cc:729] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dabeea74472746538132a520710248ea, State: Initialized, Role: LEARNER
I20240503 02:15:16.118865 13189 consensus_queue.cc:260] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:16.119382 13189 ts_tablet_manager.cc:1398] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:15:16.120059 9234 tablet_copy_service.cc:342] P cb90f34dc37b4b16b2a5b5a4ee012865: Request end of tablet copy session dabeea74472746538132a520710248ea-00fc93702514499eb32902238c46356d received from {username='slave'} at 127.23.104.202:35173
I20240503 02:15:16.120185 9234 tablet_copy_service.cc:433] P cb90f34dc37b4b16b2a5b5a4ee012865: ending tablet copy session dabeea74472746538132a520710248ea-00fc93702514499eb32902238c46356d on tablet 00fc93702514499eb32902238c46356d with peer dabeea74472746538132a520710248ea
I20240503 02:15:16.542842 11077 raft_consensus.cc:1203] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:15:16.625339 13217 raft_consensus.cc:1050] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf: attempting to promote NON_VOTER 4792082063bd4d58bc222c9c95b16335 to VOTER
I20240503 02:15:16.626554 13217 consensus_queue.cc:237] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:16.630071 11077 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:16.630455 10946 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:16.630888 13218 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:16.631280 9083 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:16.631387 13218 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:16.632011 13091 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:16.635995 8952 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:16.636168 8553 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:16.637039 8688 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:16.637221 13218 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:16.637753 8425 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:16.637980 13218 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:16.639041 13091 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:16.639768 13091 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:16.654464 8555 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.654359 13091 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.661908 8274 catalog_manager.cc:5484] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "79e41f25066e4e5abbd71e48d545c6bf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.662634 8952 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.674594 10946 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.674712 11076 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.674615 8425 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.678017 9083 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.680891 8688 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.713476 8821 consensus_queue.cc:237] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } }
I20240503 02:15:16.716279 11076 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:16.716580 8952 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:16.716607 10946 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:16.716866 13091 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:16.717129 9083 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:16.717479 13217 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:16.718171 13217 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:16.718803 13217 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:16.723919 8688 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:16.724596 13217 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:16.724985 8425 raft_consensus.cc:1261] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 79e41f25066e4e5abbd71e48d545c6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:16.725260 13091 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.726533 9083 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.727392 13232 consensus_queue.cc:1035] T 9b41db0e8b114c99ab1d8734ec8b008f P 79e41f25066e4e5abbd71e48d545c6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:16.729169 8425 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.732012 11076 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.732606 8952 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.734195 8273 catalog_manager.cc:5484] T 9b41db0e8b114c99ab1d8734ec8b008f P c399efcbba924863ac89d156dc07ac7d reported cstate change: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New cstate: current_term: 1 leader_uuid: "79e41f25066e4e5abbd71e48d545c6bf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.734284 8688 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.736388 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b41db0e8b114c99ab1d8734ec8b008f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:16.737485 10946 raft_consensus.cc:2932] T 9b41db0e8b114c99ab1d8734ec8b008f P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } }
I20240503 02:15:16.739899 8537 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b41db0e8b114c99ab1d8734ec8b008f with delete_type TABLET_DATA_TOMBSTONED (TS 2a1893ab239f4163a06bdbf8ce579431 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51278
I20240503 02:15:16.761828 13239 tablet_replica.cc:330] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431: stopping tablet replica
I20240503 02:15:16.762060 13239 raft_consensus.cc:2229] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:16.762324 13239 raft_consensus.cc:2258] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:16.768393 13239 ts_tablet_manager.cc:1875] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:16.773334 13239 ts_tablet_manager.cc:1888] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:15:16.773501 13239 log.cc:1196] T 9b41db0e8b114c99ab1d8734ec8b008f P 2a1893ab239f4163a06bdbf8ce579431: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-1/wal/wals/9b41db0e8b114c99ab1d8734ec8b008f
I20240503 02:15:16.774379 8269 catalog_manager.cc:4830] TS 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): tablet 9b41db0e8b114c99ab1d8734ec8b008f (table rebalance_test_table_0 [id=6db95e0836b1415e8dbb454f8174d455]) successfully deleted
I20240503 02:15:17.118258 13206 raft_consensus.cc:1050] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865: attempting to promote NON_VOTER dabeea74472746538132a520710248ea to VOTER
I20240503 02:15:17.119729 13206 consensus_queue.cc:237] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:17.124163 10947 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:17.124176 11076 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea [term 1 LEARNER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:17.124912 13137 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:17.125782 13137 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:17.126752 8821 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:17.127034 9083 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:17.127601 13256 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:17.128355 13256 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:17.137743 8425 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:17.138741 8688 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:17.141525 8555 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:15:17.142251 13258 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:17.145278 13206 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:17.145804 13137 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:17.161268 13256 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.161900 11076 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.165149 8425 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.165603 8688 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.166400 10947 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.167431 8821 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.167658 9083 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.171591 8273 catalog_manager.cc:5484] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 reported cstate change: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.177691 8555 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.185851 9210 consensus_queue.cc:237] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:17.188513 8821 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:17.188592 8555 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:17.188675 8688 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:17.188783 11077 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:17.188972 13258 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:17.189281 13137 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:17.189846 9083 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:17.189903 13137 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:17.190416 13206 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:17.192867 13137 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:17.193095 10947 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:15:17.193123 13257 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.193835 13266 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:15:17.195252 11077 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.195819 10947 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.196764 8269 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00fc93702514499eb32902238c46356d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:17.198525 8688 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.198580 8821 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.199344 8274 catalog_manager.cc:5484] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:15:17.200093 9083 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.207418 8555 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:17.212396 8403 tablet_service.cc:1514] Processing DeleteTablet for tablet 00fc93702514499eb32902238c46356d with delete_type TABLET_DATA_TOMBSTONED (TS 4c668b2e2d5c4edba46cbc5c36065c0c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41418
I20240503 02:15:17.213053 13270 tablet_replica.cc:330] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c: stopping tablet replica
I20240503 02:15:17.213266 13270 raft_consensus.cc:2229] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:17.213641 13270 raft_consensus.cc:2258] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:17.215730 13270 ts_tablet_manager.cc:1875] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:17.221268 13270 ts_tablet_manager.cc:1888] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:15:17.222110 13270 log.cc:1196] T 00fc93702514499eb32902238c46356d P 4c668b2e2d5c4edba46cbc5c36065c0c: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-0/wal/wals/00fc93702514499eb32902238c46356d
I20240503 02:15:17.223273 8267 catalog_manager.cc:4830] TS 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635): tablet 00fc93702514499eb32902238c46356d (table rebalance_test_table_5 [id=33a59b5afeed49d692c51024b4a1e4fd]) successfully deleted
I20240503 02:15:17.393395 9193 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:15:17.398856 8537 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:17.430001 10796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:17.437209 8801 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:15:17.437974 8403 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:17.438812 10927 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:17.457902 9062 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:17.459980 8931 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:15:17.460355 11059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:17.520591 8670 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:17.800460 8537 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:17.816711 10927 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:17.823470 8403 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:17.831286 8801 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:15:17.843714 8931 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:15:17.858091 8670 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:17.867798 9193 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:15:17.870878 9062 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:17.895344 11059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:17.929322 10796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:18.071722 8952 consensus_queue.cc:237] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:18.076018 10947 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.076426 8949 consensus_queue.cc:237] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } }
I20240503 02:15:18.076786 11077 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.077045 9210 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.077338 12711 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.077667 13337 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.078137 12711 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.079279 8555 consensus_queue.cc:237] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } }
I20240503 02:15:18.080265 9083 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.081493 8553 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.082070 12711 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.082244 8688 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.084123 13340 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.084578 13340 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:15:18.084887 8888 consensus_peers.cc:485] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 -> Peer e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200:45027): Couldn't send request to peer e512cb0d8f244ac6ab0e7150a0a8e2df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95a8f2627c5c49a39291c96eb00fec18. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:18.092363 10812 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 02:15:18.092746 8886 consensus_peers.cc:485] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 -> Peer 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747): Couldn't send request to peer 79e41f25066e4e5abbd71e48d545c6bf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:18.092926 12711 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.094444 10946 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.095078 9082 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.095613 9210 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.095942 12711 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.096477 13337 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.096761 12711 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.105154 13340 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:18.106734 10947 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:18.111512 11077 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:18.114980 8421 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.115929 13340 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.117705 8278 catalog_manager.cc:5484] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:18.173187 9081 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:18.174439 10812 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.175951 13068 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.177150 11077 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.177917 13033 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.178748 9210 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.179237 13068 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.179556 8949 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.180616 13068 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:15:18.181252 8888 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:32809, user_credentials={real_user=slave}} blocked reactor thread for 84942.2us
I20240503 02:15:18.182235 9213 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:18.188712 12711 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:18.194219 8557 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.199040 8421 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.198601 10814 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
W20240503 02:15:18.199129 8491 consensus_peers.cc:485] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 -> Peer 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): Couldn't send request to peer 846e89dd0b154d2b9bb7d4f315c9fc12. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:18.200069 13416 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.206671 8685 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:18.216804 8425 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:18.223146 13033 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } }
I20240503 02:15:18.225018 11076 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } }
I20240503 02:15:18.227425 8278 catalog_manager.cc:5484] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:18.226233 9081 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:18.232060 8949 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } }
I20240503 02:15:18.236613 8274 catalog_manager.cc:5484] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 10 to 11, NON_VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) added. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:15:18.237387 10812 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } }
I20240503 02:15:18.242091 9210 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } }
I20240503 02:15:18.255731 9083 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.256335 13416 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.215458 10944 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:18.267446 12711 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.277400 8425 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } }
I20240503 02:15:18.284835 9081 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } }
I20240503 02:15:18.299073 9212 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:18.299660 8555 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:18.308879 8688 consensus_queue.cc:237] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:18.309856 9212 consensus_queue.cc:237] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } }
I20240503 02:15:18.309710 8553 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:18.318794 10944 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.320411 13364 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.324085 8949 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.325436 9081 consensus_queue.cc:237] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: true } }
I20240503 02:15:18.326130 13364 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.331522 11077 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.332727 13364 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.352648 13364 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
W20240503 02:15:18.355001 9149 consensus_peers.cc:485] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 -> Peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): Couldn't send request to peer 99be4ff1114d4aaabcc537604fce54b3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:18.357414 8821 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.357604 10814 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.358079 13421 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.358644 8555 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.359107 9083 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.359575 13421 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.359581 12585 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.361723 11077 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.362635 13421 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:15:18.362874 9019 consensus_peers.cc:485] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d -> Peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923): Couldn't send request to peer 2a1893ab239f4163a06bdbf8ce579431. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:18.366936 13421 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.367959 8685 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.368780 11076 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:18.373458 13421 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.381553 8282 catalog_manager.cc:5484] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea reported cstate change: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New cstate: current_term: 1 leader_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:18.384984 10944 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.386255 13421 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.395911 9212 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.401942 8690 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.403308 12585 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.407538 13446 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.409154 9083 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.409672 8425 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.410200 13446 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.410810 13446 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.425976 8949 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:18.429736 9083 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:18.432829 8425 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:18.440909 10947 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
W20240503 02:15:18.454979 8625 consensus_peers.cc:485] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 -> Peer e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200:45027): Couldn't send request to peer e512cb0d8f244ac6ab0e7150a0a8e2df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe35d536ab6348c58e90d680383bb763. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:18.466903 13466 ts_tablet_manager.cc:902] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df: Initiating tablet copy from peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403)
I20240503 02:15:18.478123 13266 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:18.479053 8819 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:18.481959 13466 tablet_copy_client.cc:308] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.197:45403
I20240503 02:15:18.482775 8972 tablet_copy_service.cc:140] P 99be4ff1114d4aaabcc537604fce54b3: Received BeginTabletCopySession request for tablet 95a8f2627c5c49a39291c96eb00fec18 from peer e512cb0d8f244ac6ab0e7150a0a8e2df ({username='slave'} at 127.23.104.200:41679)
I20240503 02:15:18.482950 8972 tablet_copy_service.cc:161] P 99be4ff1114d4aaabcc537604fce54b3: Beginning new tablet copy session on tablet 95a8f2627c5c49a39291c96eb00fec18 from peer e512cb0d8f244ac6ab0e7150a0a8e2df at {username='slave'} at 127.23.104.200:41679: session id = e512cb0d8f244ac6ab0e7150a0a8e2df-95a8f2627c5c49a39291c96eb00fec18
I20240503 02:15:18.487697 8972 tablet_copy_source_session.cc:205] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:18.492642 9083 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:18.493289 8282 catalog_manager.cc:5484] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf reported cstate change: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:18.498685 10944 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:18.500998 13466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a8f2627c5c49a39291c96eb00fec18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:18.501830 13475 ts_tablet_manager.cc:902] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: Initiating tablet copy from peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403)
I20240503 02:15:18.510077 8685 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:18.518914 13466 tablet_copy_client.cc:783] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:18.519178 13466 tablet_copy_client.cc:647] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:18.519519 11077 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:18.521823 13466 tablet_copy_client.cc:523] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:18.530313 8555 raft_consensus.cc:2932] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:18.535310 13466 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap starting.
I20240503 02:15:18.537683 9212 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } }
I20240503 02:15:18.538846 9210 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.540215 12585 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.542608 8425 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.543205 12585 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.550295 8819 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.550906 12585 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.557286 11077 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.557793 12585 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.569923 12986 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: true } } }
I20240503 02:15:18.576387 8425 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: true } } }
I20240503 02:15:18.581431 10814 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: true } } }
I20240503 02:15:18.584110 8690 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: true } } }
I20240503 02:15:18.594954 8274 catalog_manager.cc:5484] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 reported cstate change: config changed from index 10 to 11, NON_VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) added. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: true } } }
I20240503 02:15:18.597244 11077 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: true } } }
I20240503 02:15:18.604300 8821 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: true } } }
I20240503 02:15:18.615249 9213 consensus_queue.cc:237] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } }
I20240503 02:15:18.624833 8555 consensus_queue.cc:237] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:18.655833 9210 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: true } } }
I20240503 02:15:18.704173 10814 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:18.705230 13068 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 02:15:18.713165 9148 consensus_peers.cc:485] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 -> Peer 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747): Couldn't send request to peer 79e41f25066e4e5abbd71e48d545c6bf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:18.715196 11077 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.716348 13474 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.726234 13466 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:18.726720 13466 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap complete.
I20240503 02:15:18.727034 13466 ts_tablet_manager.cc:1367] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent bootstrapping tablet: real 0.192s user 0.056s sys 0.003s
I20240503 02:15:18.727774 13466 raft_consensus.cc:348] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:18.728133 13466 raft_consensus.cc:729] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e512cb0d8f244ac6ab0e7150a0a8e2df, State: Initialized, Role: LEARNER
I20240503 02:15:18.728346 13466 consensus_queue.cc:260] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:18.733120 10814 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.737656 13474 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.739594 13466 ts_tablet_manager.cc:1398] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240503 02:15:18.740635 8972 tablet_copy_service.cc:342] P 99be4ff1114d4aaabcc537604fce54b3: Request end of tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-95a8f2627c5c49a39291c96eb00fec18 received from {username='slave'} at 127.23.104.200:41679
I20240503 02:15:18.740787 8972 tablet_copy_service.cc:433] P 99be4ff1114d4aaabcc537604fce54b3: ending tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-95a8f2627c5c49a39291c96eb00fec18 on tablet 95a8f2627c5c49a39291c96eb00fec18 with peer e512cb0d8f244ac6ab0e7150a0a8e2df
I20240503 02:15:18.745797 8690 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.746649 13474 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.763094 9081 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.765100 13474 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.772379 8553 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.773545 11076 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:18.774840 13033 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:18.778075 13472 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.786810 8553 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:18.786578 13474 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:18.799002 9210 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:18.799743 13033 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:18.801010 8952 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:18.801643 13033 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:18.809531 13033 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:18.812605 8952 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:18.821123 9210 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:18.856949 8282 catalog_manager.cc:5484] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 reported cstate change: config changed from index 11 to 12, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:18.874553 8952 raft_consensus.cc:1261] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:18.875115 13472 consensus_queue.cc:1035] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:18.882527 8685 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:18.890666 8282 catalog_manager.cc:5484] T 879167ff15db41a7b4ef21fa10c44b27 P 846e89dd0b154d2b9bb7d4f315c9fc12 reported cstate change: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New cstate: current_term: 1 leader_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:18.896673 10812 raft_consensus.cc:1203] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 02:15:18.902918 9083 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:18.914036 10812 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:18.925940 11076 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:18.942317 13466 ts_tablet_manager.cc:902] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df: Initiating tablet copy from peer 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495)
I20240503 02:15:18.969877 8425 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:18.983867 11077 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:19.010007 9083 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:19.023901 13475 tablet_copy_client.cc:308] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.197:45403
I20240503 02:15:19.027549 8972 tablet_copy_service.cc:140] P 99be4ff1114d4aaabcc537604fce54b3: Received BeginTabletCopySession request for tablet a7a434e2df0a4ea0b2ab5c20034033c6 from peer 79e41f25066e4e5abbd71e48d545c6bf ({username='slave'} at 127.23.104.196:50885)
I20240503 02:15:19.027716 8972 tablet_copy_service.cc:161] P 99be4ff1114d4aaabcc537604fce54b3: Beginning new tablet copy session on tablet a7a434e2df0a4ea0b2ab5c20034033c6 from peer 79e41f25066e4e5abbd71e48d545c6bf at {username='slave'} at 127.23.104.196:50885: session id = 79e41f25066e4e5abbd71e48d545c6bf-a7a434e2df0a4ea0b2ab5c20034033c6
I20240503 02:15:19.030141 13504 ts_tablet_manager.cc:902] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: Initiating tablet copy from peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:15:19.030604 8972 tablet_copy_source_session.cc:205] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:19.030663 13033 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:19.031169 13507 ts_tablet_manager.cc:902] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: Initiating tablet copy from peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:15:19.035732 13475 ts_tablet_manager.cc:1875] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240503 02:15:19.045630 8493 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.200:45027, user_credentials={real_user=slave}} blocked reactor thread for 125143us
I20240503 02:15:19.046838 8950 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:19.050101 13504 tablet_copy_client.cc:308] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.199:32809
I20240503 02:15:19.050818 9234 tablet_copy_service.cc:140] P cb90f34dc37b4b16b2a5b5a4ee012865: Received BeginTabletCopySession request for tablet 00fc93702514499eb32902238c46356d from peer 99be4ff1114d4aaabcc537604fce54b3 ({username='slave'} at 127.23.104.197:52721)
I20240503 02:15:19.050984 9234 tablet_copy_service.cc:161] P cb90f34dc37b4b16b2a5b5a4ee012865: Beginning new tablet copy session on tablet 00fc93702514499eb32902238c46356d from peer 99be4ff1114d4aaabcc537604fce54b3 at {username='slave'} at 127.23.104.197:52721: session id = 99be4ff1114d4aaabcc537604fce54b3-00fc93702514499eb32902238c46356d
I20240503 02:15:19.051576 10812 raft_consensus.cc:2932] T 879167ff15db41a7b4ef21fa10c44b27 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:19.053617 9234 tablet_copy_source_session.cc:205] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:19.057179 13033 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 02:15:19.062505 8491 consensus_peers.cc:485] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 -> Peer 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): Couldn't send request to peer 846e89dd0b154d2b9bb7d4f315c9fc12. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:19.063257 13504 ts_tablet_manager.cc:1875] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:15:19.066315 9083 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:19.067082 8421 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4792082063bd4d58bc222c9c95b16335 (127.23.104.201) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
W20240503 02:15:19.085101 8491 consensus_peers.cc:485] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 -> Peer 4792082063bd4d58bc222c9c95b16335 (127.23.104.201:46877): Couldn't send request to peer 4792082063bd4d58bc222c9c95b16335. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99e42dca2db54de5a9417df20242e2c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:19.096761 13475 ts_tablet_manager.cc:1888] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:15:19.097671 13507 tablet_copy_client.cc:308] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:41237
I20240503 02:15:19.098544 9103 tablet_copy_service.cc:140] P c399efcbba924863ac89d156dc07ac7d: Received BeginTabletCopySession request for tablet df68cc6032b14cae9e1cac074976baf8 from peer 2a1893ab239f4163a06bdbf8ce579431 ({username='slave'} at 127.23.104.194:60075)
I20240503 02:15:19.098742 9103 tablet_copy_service.cc:161] P c399efcbba924863ac89d156dc07ac7d: Beginning new tablet copy session on tablet df68cc6032b14cae9e1cac074976baf8 from peer 2a1893ab239f4163a06bdbf8ce579431 at {username='slave'} at 127.23.104.194:60075: session id = 2a1893ab239f4163a06bdbf8ce579431-df68cc6032b14cae9e1cac074976baf8
I20240503 02:15:19.102439 9103 tablet_copy_source_session.cc:205] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:19.104699 13507 ts_tablet_manager.cc:1875] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:15:19.114784 13475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a434e2df0a4ea0b2ab5c20034033c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:19.116838 13466 tablet_copy_client.cc:308] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.195:45495
I20240503 02:15:19.125671 9083 consensus_queue.cc:237] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } }
W20240503 02:15:19.135025 9018 consensus_peers.cc:485] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d -> Peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): Couldn't send request to peer 99be4ff1114d4aaabcc537604fce54b3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:19.135205 10944 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.135368 11077 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.135921 13492 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.136438 13451 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.137718 9213 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.138471 12986 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.145783 8950 consensus_queue.cc:237] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } }
I20240503 02:15:19.150696 8819 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.151248 12986 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.151540 8690 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.152381 12986 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.157454 8710 tablet_copy_service.cc:140] P 846e89dd0b154d2b9bb7d4f315c9fc12: Received BeginTabletCopySession request for tablet fe35d536ab6348c58e90d680383bb763 from peer e512cb0d8f244ac6ab0e7150a0a8e2df ({username='slave'} at 127.23.104.200:44253)
I20240503 02:15:19.157631 8710 tablet_copy_service.cc:161] P 846e89dd0b154d2b9bb7d4f315c9fc12: Beginning new tablet copy session on tablet fe35d536ab6348c58e90d680383bb763 from peer e512cb0d8f244ac6ab0e7150a0a8e2df at {username='slave'} at 127.23.104.200:44253: session id = e512cb0d8f244ac6ab0e7150a0a8e2df-fe35d536ab6348c58e90d680383bb763
I20240503 02:15:19.158248 8555 raft_consensus.cc:1261] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.160925 8710 tablet_copy_source_session.cc:205] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:19.161389 13492 consensus_queue.cc:1035] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.162487 13466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe35d536ab6348c58e90d680383bb763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:19.170185 13466 tablet_copy_client.cc:783] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:19.170497 13466 tablet_copy_client.cc:647] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:19.172389 13466 tablet_copy_client.cc:523] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:19.176014 13507 ts_tablet_manager.cc:1888] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240503 02:15:19.176347 13507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df68cc6032b14cae9e1cac074976baf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:19.177472 13546 ts_tablet_manager.cc:902] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335: Initiating tablet copy from peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923)
I20240503 02:15:19.177579 13466 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap starting.
I20240503 02:15:19.177920 13451 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:19.178597 8553 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:19.185297 13475 tablet_copy_client.cc:783] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:19.187451 13504 ts_tablet_manager.cc:1888] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:15:19.187770 13504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fc93702514499eb32902238c46356d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:19.189520 8279 catalog_manager.cc:5484] T 1a30758cbd1d4845bb9eeb02e1a2942a P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:19.191289 10947 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:19.203159 10946 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:19.203812 13340 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 02:15:19.204099 8886 consensus_peers.cc:485] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 -> Peer 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635): Couldn't send request to peer 4c668b2e2d5c4edba46cbc5c36065c0c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:19.208068 11077 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:19.207310 11076 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:19.209818 13431 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:19.212306 9212 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:19.212915 13340 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:19.226786 9210 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:19.234570 9083 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:19.234899 10814 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:19.235426 13340 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:19.235991 8553 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:19.235996 13340 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:19.237362 13337 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:19.244067 13475 tablet_copy_client.cc:647] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:19.249752 13340 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.250756 9213 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.260619 8819 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:19.260975 8553 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.263484 10944 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.269294 8279 catalog_manager.cc:5484] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 11 to 12, NON_VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) added. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.273121 13475 tablet_copy_client.cc:523] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:19.283928 13504 tablet_copy_client.cc:783] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:19.284248 13504 tablet_copy_client.cc:647] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:19.290072 13504 tablet_copy_client.cc:523] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:19.303956 13556 ts_tablet_manager.cc:902] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: Initiating tablet copy from peer cb90f34dc37b4b16b2a5b5a4ee012865 (127.23.104.199:32809)
I20240503 02:15:19.309372 13337 raft_consensus.cc:1050] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3: attempting to promote NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df to VOTER
I20240503 02:15:19.311228 13337 consensus_queue.cc:237] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } }
W20240503 02:15:19.315936 8886 consensus_peers.cc:485] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 -> Peer 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635): Couldn't send request to peer 4c668b2e2d5c4edba46cbc5c36065c0c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:19.319108 9213 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:19.319981 9081 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:19.321257 13431 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:19.321821 13431 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:19.323382 9081 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.340289 10814 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:19.347203 13340 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:19.348937 10812 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.351438 8689 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:19.352005 8690 raft_consensus.cc:2932] T 1a30758cbd1d4845bb9eeb02e1a2942a P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:19.361812 10946 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:19.363029 13340 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:19.372502 13559 ts_tablet_manager.cc:902] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Initiating tablet copy from peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923)
I20240503 02:15:19.374644 9082 consensus_queue.cc:237] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:19.376389 8553 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:19.377190 11076 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:19.377689 13560 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:19.378125 8685 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:19.378491 10946 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.379608 13546 tablet_copy_client.cc:308] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:45923
I20240503 02:15:19.380048 13560 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:19.381649 11077 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.383213 13492 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.384061 8555 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.384534 13492 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:15:19.387449 9018 consensus_peers.cc:485] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d -> Peer dabeea74472746538132a520710248ea (127.23.104.202:37759): Couldn't send request to peer dabeea74472746538132a520710248ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5028c92b346845c38b3356a0d8c7507f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:19.394539 10812 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.395164 13492 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.396060 13560 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.396893 8553 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.399070 9212 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.399561 13492 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.402673 8950 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.403630 13431 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:19.403623 13492 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.404695 8690 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.405166 13492 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.406152 9212 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.408006 8685 raft_consensus.cc:858] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } }:
New config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } }
I20240503 02:15:19.408499 9081 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
W20240503 02:15:19.421988 8685 raft_consensus.cc:1565] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 13' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } }:
New config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } }
I20240503 02:15:19.429054 8282 catalog_manager.cc:5484] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 12 to 13, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.431401 10814 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.432574 13466 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:19.433084 13466 tablet_bootstrap.cc:492] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df: Bootstrap complete.
I20240503 02:15:19.433369 13466 ts_tablet_manager.cc:1367] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent bootstrapping tablet: real 0.257s user 0.069s sys 0.037s
I20240503 02:15:19.434271 13466 raft_consensus.cc:348] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:19.434643 13466 raft_consensus.cc:729] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e512cb0d8f244ac6ab0e7150a0a8e2df, State: Initialized, Role: LEARNER
I20240503 02:15:19.434902 13466 consensus_queue.cc:260] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } }
I20240503 02:15:19.435676 13466 ts_tablet_manager.cc:1398] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:15:19.436664 8710 tablet_copy_service.cc:342] P 846e89dd0b154d2b9bb7d4f315c9fc12: Request end of tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-fe35d536ab6348c58e90d680383bb763 received from {username='slave'} at 127.23.104.200:44253
I20240503 02:15:19.436803 8710 tablet_copy_service.cc:433] P 846e89dd0b154d2b9bb7d4f315c9fc12: ending tablet copy session e512cb0d8f244ac6ab0e7150a0a8e2df-fe35d536ab6348c58e90d680383bb763 on tablet fe35d536ab6348c58e90d680383bb763 with peer e512cb0d8f244ac6ab0e7150a0a8e2df
I20240503 02:15:19.437178 13507 tablet_copy_client.cc:783] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:19.437510 13507 tablet_copy_client.cc:647] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:19.447093 11077 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.455982 10944 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.459707 13475 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:15:19.464594 13556 tablet_copy_client.cc:308] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.199:32809
I20240503 02:15:19.465377 9234 tablet_copy_service.cc:140] P cb90f34dc37b4b16b2a5b5a4ee012865: Received BeginTabletCopySession request for tablet 879167ff15db41a7b4ef21fa10c44b27 from peer 79e41f25066e4e5abbd71e48d545c6bf ({username='slave'} at 127.23.104.196:58583)
I20240503 02:15:19.465561 9234 tablet_copy_service.cc:161] P cb90f34dc37b4b16b2a5b5a4ee012865: Beginning new tablet copy session on tablet 879167ff15db41a7b4ef21fa10c44b27 from peer 79e41f25066e4e5abbd71e48d545c6bf at {username='slave'} at 127.23.104.196:58583: session id = 79e41f25066e4e5abbd71e48d545c6bf-879167ff15db41a7b4ef21fa10c44b27
I20240503 02:15:19.467629 13504 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:15:19.468536 9234 tablet_copy_source_session.cc:205] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:19.469725 13556 ts_tablet_manager.cc:1875] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:15:19.473846 13492 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:19.478780 13556 ts_tablet_manager.cc:1888] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:15:19.480888 13556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879167ff15db41a7b4ef21fa10c44b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:19.480463 8685 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: NON_VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.485605 10942 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:19.488909 13556 tablet_copy_client.cc:783] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:19.489284 13556 tablet_copy_client.cc:647] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:19.491099 13507 tablet_copy_client.cc:523] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:19.491674 13556 tablet_copy_client.cc:523] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:19.493732 8950 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:19.496326 8278 catalog_manager.cc:5484] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335 reported cstate change: config changed from index 10 to 11, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:19.496886 13507 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap starting.
I20240503 02:15:19.497437 13556 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:15:19.504442 9212 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:19.504590 10814 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:19.516944 8553 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:19.527312 10812 raft_consensus.cc:1203] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 02:15:19.528494 8689 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER dabeea74472746538132a520710248ea (127.23.104.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } } }
I20240503 02:15:19.533965 8578 tablet_copy_service.cc:140] P 2a1893ab239f4163a06bdbf8ce579431: Received BeginTabletCopySession request for tablet 99e42dca2db54de5a9417df20242e2c2 from peer 4792082063bd4d58bc222c9c95b16335 ({username='slave'} at 127.23.104.201:54817)
I20240503 02:15:19.534101 8578 tablet_copy_service.cc:161] P 2a1893ab239f4163a06bdbf8ce579431: Beginning new tablet copy session on tablet 99e42dca2db54de5a9417df20242e2c2 from peer 4792082063bd4d58bc222c9c95b16335 at {username='slave'} at 127.23.104.201:54817: session id = 4792082063bd4d58bc222c9c95b16335-99e42dca2db54de5a9417df20242e2c2
I20240503 02:15:19.536983 8578 tablet_copy_source_session.cc:205] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:19.540688 13546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e42dca2db54de5a9417df20242e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:19.548164 13546 tablet_copy_client.cc:783] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:19.548418 13546 tablet_copy_client.cc:647] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:19.554291 13546 tablet_copy_client.cc:523] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:19.554174 8685 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.557987 13546 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335: Bootstrap starting.
I20240503 02:15:19.560094 13559 tablet_copy_client.cc:308] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:45923
I20240503 02:15:19.561064 8578 tablet_copy_service.cc:140] P 2a1893ab239f4163a06bdbf8ce579431: Received BeginTabletCopySession request for tablet 99e42dca2db54de5a9417df20242e2c2 from peer 846e89dd0b154d2b9bb7d4f315c9fc12 ({username='slave'} at 127.23.104.195:48897)
I20240503 02:15:19.561245 8578 tablet_copy_service.cc:161] P 2a1893ab239f4163a06bdbf8ce579431: Beginning new tablet copy session on tablet 99e42dca2db54de5a9417df20242e2c2 from peer 846e89dd0b154d2b9bb7d4f315c9fc12 at {username='slave'} at 127.23.104.195:48897: session id = 846e89dd0b154d2b9bb7d4f315c9fc12-99e42dca2db54de5a9417df20242e2c2
I20240503 02:15:19.564852 8578 tablet_copy_source_session.cc:205] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:19.566560 13559 ts_tablet_manager.cc:1875] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:15:19.567370 8950 consensus_queue.cc:237] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } }
I20240503 02:15:19.571856 10812 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
W20240503 02:15:19.572459 8886 consensus_peers.cc:485] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 -> Peer 4c668b2e2d5c4edba46cbc5c36065c0c (127.23.104.193:38635): Couldn't send request to peer 4c668b2e2d5c4edba46cbc5c36065c0c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:19.572595 13337 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:15:19.574400 11077 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 02:15:19.574518 9213 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 02:15:19.575078 13573 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:15:19.575217 8553 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 02:15:19.575858 13560 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:15:19.576299 13560 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:15:19.577517 10944 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 02:15:19.578140 13573 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:15:19.582643 9081 raft_consensus.cc:1261] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 02:15:19.583454 13573 consensus_queue.cc:1035] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:15:19.584345 13573 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.591143 8553 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.595358 13559 ts_tablet_manager.cc:1888] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240503 02:15:19.595693 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e42dca2db54de5a9417df20242e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:19.598057 8274 catalog_manager.cc:5484] T 95a8f2627c5c49a39291c96eb00fec18 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 13 to 14, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.601306 10944 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.601517 9081 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.604382 8268 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 95a8f2627c5c49a39291c96eb00fec18 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:19.605474 11077 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.608006 8670 tablet_service.cc:1514] Processing DeleteTablet for tablet 95a8f2627c5c49a39291c96eb00fec18 with delete_type TABLET_DATA_TOMBSTONED (TS 846e89dd0b154d2b9bb7d4f315c9fc12 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:36802
I20240503 02:15:19.616166 9213 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.616987 10814 raft_consensus.cc:2932] T 95a8f2627c5c49a39291c96eb00fec18 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: NON_VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { promote: true } } }
I20240503 02:15:19.636646 13602 ts_tablet_manager.cc:902] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: Initiating tablet copy from peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:15:19.644760 13559 tablet_copy_client.cc:783] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:19.645157 13559 tablet_copy_client.cc:647] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:19.650008 13559 tablet_copy_client.cc:523] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:19.658126 8950 consensus_queue.cc:237] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } }
I20240503 02:15:19.658833 13601 tablet_replica.cc:330] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12: stopping tablet replica
I20240503 02:15:19.659101 13601 raft_consensus.cc:2229] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:19.659641 13601 raft_consensus.cc:2258] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:19.662305 13601 ts_tablet_manager.cc:1875] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:19.665549 13559 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap starting.
W20240503 02:15:19.677131 8886 consensus_peers.cc:485] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 -> Peer 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747): Couldn't send request to peer 79e41f25066e4e5abbd71e48d545c6bf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:19.678086 10944 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.678714 13573 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.687229 11077 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.688019 13573 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.692945 9213 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.693718 13573 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.694113 9081 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.695008 8685 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.696313 13431 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.696415 8425 raft_consensus.cc:1261] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:19.697124 13431 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.710937 13431 consensus_queue.cc:1035] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:19.743043 13560 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:19.744199 11077 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:19.750443 8279 catalog_manager.cc:5484] T 64c52f9977ae4eb4ac803be99b6426c2 P dabeea74472746538132a520710248ea reported cstate change: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:19.753449 9212 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:19.764061 10944 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:19.764196 8421 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:19.769012 9083 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:19.777693 13602 tablet_copy_client.cc:308] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:41237
I20240503 02:15:19.782291 9103 tablet_copy_service.cc:140] P c399efcbba924863ac89d156dc07ac7d: Received BeginTabletCopySession request for tablet 1a30758cbd1d4845bb9eeb02e1a2942a from peer 99be4ff1114d4aaabcc537604fce54b3 ({username='slave'} at 127.23.104.197:48207)
I20240503 02:15:19.782461 9103 tablet_copy_service.cc:161] P c399efcbba924863ac89d156dc07ac7d: Beginning new tablet copy session on tablet 1a30758cbd1d4845bb9eeb02e1a2942a from peer 99be4ff1114d4aaabcc537604fce54b3 at {username='slave'} at 127.23.104.197:48207: session id = 99be4ff1114d4aaabcc537604fce54b3-1a30758cbd1d4845bb9eeb02e1a2942a
I20240503 02:15:19.785634 9103 tablet_copy_source_session.cc:205] T 1a30758cbd1d4845bb9eeb02e1a2942a P c399efcbba924863ac89d156dc07ac7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:19.789626 13602 ts_tablet_manager.cc:1875] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:15:19.793998 13627 ts_tablet_manager.cc:902] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea: Initiating tablet copy from peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:15:19.801988 13602 ts_tablet_manager.cc:1888] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:15:19.802275 13602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a30758cbd1d4845bb9eeb02e1a2942a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:19.829998 13601 ts_tablet_manager.cc:1888] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20240503 02:15:19.830181 13601 log.cc:1196] T 95a8f2627c5c49a39291c96eb00fec18 P 846e89dd0b154d2b9bb7d4f315c9fc12: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/95a8f2627c5c49a39291c96eb00fec18
I20240503 02:15:19.832971 8267 catalog_manager.cc:4830] TS 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): tablet 95a8f2627c5c49a39291c96eb00fec18 (table rebalance_test_table_3 [id=5f8fb8023f5e4aa087128fbb3c6b637f]) successfully deleted
I20240503 02:15:19.836505 8689 raft_consensus.cc:2932] T 64c52f9977ae4eb4ac803be99b6426c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:19.847050 13602 tablet_copy_client.cc:783] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:19.847313 13602 tablet_copy_client.cc:647] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:19.860591 13602 tablet_copy_client.cc:523] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:19.877362 13627 tablet_copy_client.cc:308] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.198:41237
I20240503 02:15:19.878015 9103 tablet_copy_service.cc:140] P c399efcbba924863ac89d156dc07ac7d: Received BeginTabletCopySession request for tablet 5028c92b346845c38b3356a0d8c7507f from peer dabeea74472746538132a520710248ea ({username='slave'} at 127.23.104.202:46779)
W20240503 02:15:19.878120 13546 log.cc:924] Time spent T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335: Append to log took a long time: real 0.050s user 0.003s sys 0.000s
I20240503 02:15:19.878219 9103 tablet_copy_service.cc:161] P c399efcbba924863ac89d156dc07ac7d: Beginning new tablet copy session on tablet 5028c92b346845c38b3356a0d8c7507f from peer dabeea74472746538132a520710248ea at {username='slave'} at 127.23.104.202:46779: session id = dabeea74472746538132a520710248ea-5028c92b346845c38b3356a0d8c7507f
I20240503 02:15:19.878458 13507 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:19.878877 13507 tablet_bootstrap.cc:492] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: Bootstrap complete.
I20240503 02:15:19.879158 13507 ts_tablet_manager.cc:1367] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: Time spent bootstrapping tablet: real 0.383s user 0.061s sys 0.000s
I20240503 02:15:19.880046 13507 raft_consensus.cc:348] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: true } }
I20240503 02:15:19.880381 13507 raft_consensus.cc:729] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a1893ab239f4163a06bdbf8ce579431, State: Initialized, Role: LEARNER
I20240503 02:15:19.880610 13507 consensus_queue.cc:260] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: NON_VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: true } }
I20240503 02:15:19.881395 13507 ts_tablet_manager.cc:1398] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:15:19.881973 13602 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap starting.
I20240503 02:15:19.882123 9102 tablet_copy_service.cc:342] P c399efcbba924863ac89d156dc07ac7d: Request end of tablet copy session 2a1893ab239f4163a06bdbf8ce579431-df68cc6032b14cae9e1cac074976baf8 received from {username='slave'} at 127.23.104.194:60075
I20240503 02:15:19.882243 9103 tablet_copy_source_session.cc:205] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:19.882261 9102 tablet_copy_service.cc:433] P c399efcbba924863ac89d156dc07ac7d: ending tablet copy session 2a1893ab239f4163a06bdbf8ce579431-df68cc6032b14cae9e1cac074976baf8 on tablet df68cc6032b14cae9e1cac074976baf8 with peer 2a1893ab239f4163a06bdbf8ce579431
I20240503 02:15:19.884631 13475 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:19.885136 13475 tablet_bootstrap.cc:492] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap complete.
I20240503 02:15:19.885535 13475 ts_tablet_manager.cc:1367] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.426s user 0.053s sys 0.008s
I20240503 02:15:19.886363 13475 raft_consensus.cc:348] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } }
I20240503 02:15:19.886718 13627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5028c92b346845c38b3356a0d8c7507f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:19.886735 13475 raft_consensus.cc:729] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: LEARNER
I20240503 02:15:19.886976 13475 consensus_queue.cc:260] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } }
I20240503 02:15:19.887777 13475 ts_tablet_manager.cc:1398] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:15:19.889467 8972 tablet_copy_service.cc:342] P 99be4ff1114d4aaabcc537604fce54b3: Request end of tablet copy session 79e41f25066e4e5abbd71e48d545c6bf-a7a434e2df0a4ea0b2ab5c20034033c6 received from {username='slave'} at 127.23.104.196:50885
I20240503 02:15:19.889585 8972 tablet_copy_service.cc:433] P 99be4ff1114d4aaabcc537604fce54b3: ending tablet copy session 79e41f25066e4e5abbd71e48d545c6bf-a7a434e2df0a4ea0b2ab5c20034033c6 on tablet a7a434e2df0a4ea0b2ab5c20034033c6 with peer 79e41f25066e4e5abbd71e48d545c6bf
I20240503 02:15:19.916507 8553 consensus_queue.cc:237] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } }
I20240503 02:15:19.920938 13627 tablet_copy_client.cc:783] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:19.921265 13627 tablet_copy_client.cc:647] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:19.924073 13627 tablet_copy_client.cc:523] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 02:15:19.934170 8491 consensus_peers.cc:485] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 -> Peer 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196:34747): Couldn't send request to peer 79e41f25066e4e5abbd71e48d545c6bf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:19.935230 13559 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:19.935751 13559 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Bootstrap complete.
I20240503 02:15:19.936079 13559 ts_tablet_manager.cc:1367] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent bootstrapping tablet: real 0.271s user 0.048s sys 0.016s
I20240503 02:15:19.936861 13559 raft_consensus.cc:348] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:19.937217 13559 raft_consensus.cc:729] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 846e89dd0b154d2b9bb7d4f315c9fc12, State: Initialized, Role: LEARNER
I20240503 02:15:19.937460 13559 consensus_queue.cc:260] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:19.939231 13559 ts_tablet_manager.cc:1398] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:15:19.939903 8578 tablet_copy_service.cc:342] P 2a1893ab239f4163a06bdbf8ce579431: Request end of tablet copy session 846e89dd0b154d2b9bb7d4f315c9fc12-99e42dca2db54de5a9417df20242e2c2 received from {username='slave'} at 127.23.104.195:48897
I20240503 02:15:19.940018 8578 tablet_copy_service.cc:433] P 2a1893ab239f4163a06bdbf8ce579431: ending tablet copy session 846e89dd0b154d2b9bb7d4f315c9fc12-99e42dca2db54de5a9417df20242e2c2 on tablet 99e42dca2db54de5a9417df20242e2c2 with peer 846e89dd0b154d2b9bb7d4f315c9fc12
I20240503 02:15:19.941852 8685 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:19.943022 8425 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:19.943847 9081 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:19.944541 8950 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:19.945726 13068 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:19.945968 13033 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:19.946923 13517 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:19.947410 9213 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:19.948895 10814 raft_consensus.cc:1261] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:15:19.951104 13517 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:19.952289 13068 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:19.960117 13627 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea: Bootstrap starting.
W20240503 02:15:19.995674 8491 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:41237, user_credentials={real_user=slave}} blocked reactor thread for 45041.8us
I20240503 02:15:20.027995 8553 raft_consensus.cc:1203] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 02:15:20.028707 13033 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:20.031661 13556 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:20.031944 13504 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:20.032471 13504 tablet_bootstrap.cc:492] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap complete.
I20240503 02:15:20.033001 13504 ts_tablet_manager.cc:1367] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.565s user 0.059s sys 0.000s
I20240503 02:15:20.033784 13504 raft_consensus.cc:348] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } }
I20240503 02:15:20.034091 13504 raft_consensus.cc:729] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: LEARNER
I20240503 02:15:20.034323 13504 consensus_queue.cc:260] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } }
I20240503 02:15:20.038328 13504 ts_tablet_manager.cc:1398] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:15:20.037997 9081 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:20.039198 8950 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:20.046161 8279 catalog_manager.cc:5484] T 4679ab055c1a40f19bf7b602451044f6 P c399efcbba924863ac89d156dc07ac7d reported cstate change: config changed from index 7 to 8, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New cstate: current_term: 1 leader_uuid: "2a1893ab239f4163a06bdbf8ce579431" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:20.049715 8689 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:20.064661 9234 tablet_copy_service.cc:342] P cb90f34dc37b4b16b2a5b5a4ee012865: Request end of tablet copy session 99be4ff1114d4aaabcc537604fce54b3-00fc93702514499eb32902238c46356d received from {username='slave'} at 127.23.104.197:52721
I20240503 02:15:20.064851 9234 tablet_copy_service.cc:433] P cb90f34dc37b4b16b2a5b5a4ee012865: ending tablet copy session 99be4ff1114d4aaabcc537604fce54b3-00fc93702514499eb32902238c46356d on tablet 00fc93702514499eb32902238c46356d with peer 99be4ff1114d4aaabcc537604fce54b3
W20240503 02:15:20.084160 13655 log.cc:924] Time spent T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240503 02:15:20.088829 8425 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:20.090773 13556 tablet_bootstrap.cc:492] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap complete.
I20240503 02:15:20.091114 13556 ts_tablet_manager.cc:1367] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.594s user 0.058s sys 0.007s
I20240503 02:15:20.092005 13460 raft_consensus.cc:1050] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12: attempting to promote NON_VOTER e512cb0d8f244ac6ab0e7150a0a8e2df to VOTER
I20240503 02:15:20.093152 13460 consensus_queue.cc:237] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:20.103988 13556 raft_consensus.cc:348] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } }
I20240503 02:15:20.104467 13556 raft_consensus.cc:729] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: LEARNER
I20240503 02:15:20.104780 13556 consensus_queue.cc:260] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } }
I20240503 02:15:20.105836 10814 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 LEARNER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.106451 13556 ts_tablet_manager.cc:1398] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240503 02:15:20.107210 9212 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.107221 9234 tablet_copy_service.cc:342] P cb90f34dc37b4b16b2a5b5a4ee012865: Request end of tablet copy session 79e41f25066e4e5abbd71e48d545c6bf-879167ff15db41a7b4ef21fa10c44b27 received from {username='slave'} at 127.23.104.196:58583
I20240503 02:15:20.108140 9234 tablet_copy_service.cc:433] P cb90f34dc37b4b16b2a5b5a4ee012865: ending tablet copy session 79e41f25066e4e5abbd71e48d545c6bf-879167ff15db41a7b4ef21fa10c44b27 on tablet 879167ff15db41a7b4ef21fa10c44b27 with peer 79e41f25066e4e5abbd71e48d545c6bf
I20240503 02:15:20.108103 13661 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.109592 8950 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.111936 10944 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.120975 13661 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.130456 8821 raft_consensus.cc:1203] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 02:15:20.131985 13460 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.132704 11076 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.133731 9081 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.134228 13667 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.134464 8425 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.135250 13661 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.136152 13661 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.139634 13661 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.158643 13667 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.158766 10812 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.164636 9212 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.164671 11077 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.171689 8421 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.172482 8278 catalog_manager.cc:5484] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 reported cstate change: config changed from index 11 to 12, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.185184 8950 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.188601 9083 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.197273 13556 ts_tablet_manager.cc:902] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: Initiating tablet copy from peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403)
W20240503 02:15:20.209475 8493 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.199:32809, user_credentials={real_user=slave}} blocked reactor thread for 260900us
I20240503 02:15:20.216506 8821 raft_consensus.cc:1203] T 879167ff15db41a7b4ef21fa10c44b27 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 02:15:20.227995 13517 consensus_queue.cc:1035] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:20.237771 13546 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:20.238260 13546 tablet_bootstrap.cc:492] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335: Bootstrap complete.
I20240503 02:15:20.238552 13546 ts_tablet_manager.cc:1367] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335: Time spent bootstrapping tablet: real 0.681s user 0.065s sys 0.002s
I20240503 02:15:20.239323 13546 raft_consensus.cc:348] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:20.239670 13546 raft_consensus.cc:729] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4792082063bd4d58bc222c9c95b16335, State: Initialized, Role: LEARNER
I20240503 02:15:20.253736 9083 consensus_queue.cc:237] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } }
I20240503 02:15:20.258962 8689 consensus_queue.cc:237] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } }
I20240503 02:15:20.259953 9212 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:20.260536 10947 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.263157 10814 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.263319 9213 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.263942 13460 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.264434 13460 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.266991 9082 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.267071 8421 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.267421 8950 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.267984 13667 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.268391 13667 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.268834 13667 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.272868 13667 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER dabeea74472746538132a520710248ea (127.23.104.202) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.273171 10812 raft_consensus.cc:2932] T 4679ab055c1a40f19bf7b602451044f6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } } }
I20240503 02:15:20.276531 8421 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER dabeea74472746538132a520710248ea (127.23.104.202) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.281329 13627 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:20.281782 13627 tablet_bootstrap.cc:492] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea: Bootstrap complete.
I20240503 02:15:20.282099 13627 ts_tablet_manager.cc:1367] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea: Time spent bootstrapping tablet: real 0.322s user 0.057s sys 0.001s
I20240503 02:15:20.282866 13627 raft_consensus.cc:348] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:20.282850 8282 catalog_manager.cc:5484] T fe35d536ab6348c58e90d680383bb763 P 4c668b2e2d5c4edba46cbc5c36065c0c reported cstate change: config changed from index 12 to 13, VOTER dabeea74472746538132a520710248ea (127.23.104.202) evicted. New cstate: current_term: 1 leader_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.283202 13627 raft_consensus.cc:729] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dabeea74472746538132a520710248ea, State: Initialized, Role: LEARNER
I20240503 02:15:20.283421 13627 consensus_queue.cc:260] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: NON_VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: true } }
I20240503 02:15:20.287320 9082 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER dabeea74472746538132a520710248ea (127.23.104.202) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.290868 13682 ts_tablet_manager.cc:902] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: Initiating tablet copy from peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403)
I20240503 02:15:20.290776 13627 ts_tablet_manager.cc:1398] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240503 02:15:20.291916 9103 tablet_copy_service.cc:342] P c399efcbba924863ac89d156dc07ac7d: Request end of tablet copy session dabeea74472746538132a520710248ea-5028c92b346845c38b3356a0d8c7507f received from {username='slave'} at 127.23.104.202:46779
I20240503 02:15:20.292054 9103 tablet_copy_service.cc:433] P c399efcbba924863ac89d156dc07ac7d: ending tablet copy session dabeea74472746538132a520710248ea-5028c92b346845c38b3356a0d8c7507f on tablet 5028c92b346845c38b3356a0d8c7507f with peer dabeea74472746538132a520710248ea
I20240503 02:15:20.295138 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe35d536ab6348c58e90d680383bb763 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:20.297255 10814 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER dabeea74472746538132a520710248ea (127.23.104.202) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.298334 13546 consensus_queue.cc:260] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: NON_VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:20.298449 8950 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER dabeea74472746538132a520710248ea (127.23.104.202) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.305588 8689 raft_consensus.cc:1203] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240503 02:15:20.307834 9212 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER dabeea74472746538132a520710248ea (127.23.104.202) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.312649 13556 tablet_copy_client.cc:308] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.197:45403
I20240503 02:15:20.313786 8972 tablet_copy_service.cc:140] P 99be4ff1114d4aaabcc537604fce54b3: Received BeginTabletCopySession request for tablet 64c52f9977ae4eb4ac803be99b6426c2 from peer 79e41f25066e4e5abbd71e48d545c6bf ({username='slave'} at 127.23.104.196:50885)
I20240503 02:15:20.313982 8972 tablet_copy_service.cc:161] P 99be4ff1114d4aaabcc537604fce54b3: Beginning new tablet copy session on tablet 64c52f9977ae4eb4ac803be99b6426c2 from peer 79e41f25066e4e5abbd71e48d545c6bf at {username='slave'} at 127.23.104.196:50885: session id = 79e41f25066e4e5abbd71e48d545c6bf-64c52f9977ae4eb4ac803be99b6426c2
I20240503 02:15:20.315487 8950 raft_consensus.cc:1203] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 02:15:20.318702 8972 tablet_copy_source_session.cc:205] T 64c52f9977ae4eb4ac803be99b6426c2 P 99be4ff1114d4aaabcc537604fce54b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:20.319463 11077 raft_consensus.cc:1203] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 02:15:20.319962 13556 ts_tablet_manager.cc:1875] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:15:20.322605 13546 ts_tablet_manager.cc:1398] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335: Time spent starting tablet: real 0.084s user 0.004s sys 0.000s
I20240503 02:15:20.323370 8578 tablet_copy_service.cc:342] P 2a1893ab239f4163a06bdbf8ce579431: Request end of tablet copy session 4792082063bd4d58bc222c9c95b16335-99e42dca2db54de5a9417df20242e2c2 received from {username='slave'} at 127.23.104.201:54817
I20240503 02:15:20.323598 8578 tablet_copy_service.cc:433] P 2a1893ab239f4163a06bdbf8ce579431: ending tablet copy session 4792082063bd4d58bc222c9c95b16335-99e42dca2db54de5a9417df20242e2c2 on tablet 99e42dca2db54de5a9417df20242e2c2 with peer 4792082063bd4d58bc222c9c95b16335
W20240503 02:15:20.325968 9018 consensus_peers.cc:485] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d -> Peer 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197:45403): Couldn't send request to peer 99be4ff1114d4aaabcc537604fce54b3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:20.326972 10814 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:20.327528 12585 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:20.334324 8425 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:20.334869 9212 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:20.335083 10946 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:20.335278 8821 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:20.335765 12585 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:20.336019 8553 raft_consensus.cc:1261] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:15:20.336313 12585 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:20.336788 12585 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:20.338536 13492 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:20.339135 13492 consensus_queue.cc:1035] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:15:20.346669 10946 raft_consensus.cc:1203] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240503 02:15:20.358304 10944 raft_consensus.cc:1261] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 846e89dd0b154d2b9bb7d4f315c9fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.359197 13681 consensus_queue.cc:1035] T fe35d536ab6348c58e90d680383bb763 P 846e89dd0b154d2b9bb7d4f315c9fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.361291 10944 raft_consensus.cc:2932] T fe35d536ab6348c58e90d680383bb763 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER dabeea74472746538132a520710248ea (127.23.104.202) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } }
I20240503 02:15:20.378537 13602 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:20.378733 13556 ts_tablet_manager.cc:1888] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:15:20.378935 13602 tablet_bootstrap.cc:492] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: Bootstrap complete.
I20240503 02:15:20.378891 11059 tablet_service.cc:1514] Processing DeleteTablet for tablet fe35d536ab6348c58e90d680383bb763 with delete_type TABLET_DATA_TOMBSTONED (TS dabeea74472746538132a520710248ea not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:57958
I20240503 02:15:20.379027 13556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c52f9977ae4eb4ac803be99b6426c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:20.379166 13602 ts_tablet_manager.cc:1367] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: Time spent bootstrapping tablet: real 0.497s user 0.033s sys 0.024s
I20240503 02:15:20.379940 13707 tablet_replica.cc:330] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea: stopping tablet replica
I20240503 02:15:20.380203 13682 tablet_copy_client.cc:308] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.197:45403
I20240503 02:15:20.380734 13707 raft_consensus.cc:2229] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:20.381853 13707 raft_consensus.cc:2258] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:20.381970 13556 tablet_copy_client.cc:783] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:20.382196 13556 tablet_copy_client.cc:647] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:20.383261 13475 ts_tablet_manager.cc:902] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: Initiating tablet copy from peer 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194:45923)
I20240503 02:15:20.385679 13707 ts_tablet_manager.cc:1875] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:20.386476 13556 tablet_copy_client.cc:523] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:20.388074 8972 tablet_copy_service.cc:140] P 99be4ff1114d4aaabcc537604fce54b3: Received BeginTabletCopySession request for tablet 95a8f2627c5c49a39291c96eb00fec18 from peer 4c668b2e2d5c4edba46cbc5c36065c0c ({username='slave'} at 127.23.104.193:53839)
I20240503 02:15:20.388278 8972 tablet_copy_service.cc:161] P 99be4ff1114d4aaabcc537604fce54b3: Beginning new tablet copy session on tablet 95a8f2627c5c49a39291c96eb00fec18 from peer 4c668b2e2d5c4edba46cbc5c36065c0c at {username='slave'} at 127.23.104.193:53839: session id = 4c668b2e2d5c4edba46cbc5c36065c0c-95a8f2627c5c49a39291c96eb00fec18
I20240503 02:15:20.389966 13556 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:15:20.395408 8972 tablet_copy_source_session.cc:205] T 95a8f2627c5c49a39291c96eb00fec18 P 99be4ff1114d4aaabcc537604fce54b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:20.401619 13707 ts_tablet_manager.cc:1888] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240503 02:15:20.400990 13492 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:20.401865 13707 log.cc:1196] T fe35d536ab6348c58e90d680383bb763 P dabeea74472746538132a520710248ea: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-9/wal/wals/fe35d536ab6348c58e90d680383bb763
I20240503 02:15:20.402953 8268 catalog_manager.cc:4830] TS dabeea74472746538132a520710248ea (127.23.104.202:37759): tablet fe35d536ab6348c58e90d680383bb763 (table rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]) successfully deleted
I20240503 02:15:20.404737 13682 ts_tablet_manager.cc:1875] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:15:20.404426 8555 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:20.414768 8273 catalog_manager.cc:5484] T b71357f9f8ef449189f485b3246869f1 P 2a1893ab239f4163a06bdbf8ce579431 reported cstate change: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:20.421380 10812 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:20.421911 9212 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
W20240503 02:15:20.426172 9017 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.201:46877, user_credentials={real_user=slave}} blocked reactor thread for 89695us
I20240503 02:15:20.434746 13682 ts_tablet_manager.cc:1888] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240503 02:15:20.435083 13682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a8f2627c5c49a39291c96eb00fec18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:20.442157 8821 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:20.449169 13602 raft_consensus.cc:348] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } }
I20240503 02:15:20.450783 13602 raft_consensus.cc:729] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99be4ff1114d4aaabcc537604fce54b3, State: Initialized, Role: LEARNER
I20240503 02:15:20.451602 13602 consensus_queue.cc:260] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } }
I20240503 02:15:20.455988 13602 ts_tablet_manager.cc:1398] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3: Time spent starting tablet: real 0.077s user 0.006s sys 0.000s
I20240503 02:15:20.458052 9103 tablet_copy_service.cc:342] P c399efcbba924863ac89d156dc07ac7d: Request end of tablet copy session 99be4ff1114d4aaabcc537604fce54b3-1a30758cbd1d4845bb9eeb02e1a2942a received from {username='slave'} at 127.23.104.197:48207
I20240503 02:15:20.458214 9103 tablet_copy_service.cc:433] P c399efcbba924863ac89d156dc07ac7d: ending tablet copy session 99be4ff1114d4aaabcc537604fce54b3-1a30758cbd1d4845bb9eeb02e1a2942a on tablet 1a30758cbd1d4845bb9eeb02e1a2942a with peer 99be4ff1114d4aaabcc537604fce54b3
I20240503 02:15:20.462322 10947 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:20.471076 13682 tablet_copy_client.cc:783] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:20.472018 13682 tablet_copy_client.cc:647] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:20.472275 8425 raft_consensus.cc:2932] T b71357f9f8ef449189f485b3246869f1 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 (127.23.104.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: NON_VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: true } } }
I20240503 02:15:20.477896 13682 tablet_copy_client.cc:523] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:20.490613 13682 tablet_bootstrap.cc:492] T 95a8f2627c5c49a39291c96eb00fec18 P 4c668b2e2d5c4edba46cbc5c36065c0c: Bootstrap starting.
I20240503 02:15:20.492508 13475 tablet_copy_client.cc:308] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Beginning tablet copy session from remote peer at address 127.23.104.194:45923
I20240503 02:15:20.493245 8578 tablet_copy_service.cc:140] P 2a1893ab239f4163a06bdbf8ce579431: Received BeginTabletCopySession request for tablet 4679ab055c1a40f19bf7b602451044f6 from peer 79e41f25066e4e5abbd71e48d545c6bf ({username='slave'} at 127.23.104.196:40793)
I20240503 02:15:20.493423 8578 tablet_copy_service.cc:161] P 2a1893ab239f4163a06bdbf8ce579431: Beginning new tablet copy session on tablet 4679ab055c1a40f19bf7b602451044f6 from peer 79e41f25066e4e5abbd71e48d545c6bf at {username='slave'} at 127.23.104.196:40793: session id = 79e41f25066e4e5abbd71e48d545c6bf-4679ab055c1a40f19bf7b602451044f6
I20240503 02:15:20.497330 8578 tablet_copy_source_session.cc:205] T 4679ab055c1a40f19bf7b602451044f6 P 2a1893ab239f4163a06bdbf8ce579431: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:20.498612 13475 ts_tablet_manager.cc:1875] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:15:20.501020 13556 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:20.501478 13556 tablet_bootstrap.cc:492] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap complete.
I20240503 02:15:20.501732 13556 ts_tablet_manager.cc:1367] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.112s user 0.064s sys 0.005s
I20240503 02:15:20.502593 13556 raft_consensus.cc:348] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } }
I20240503 02:15:20.502934 13556 raft_consensus.cc:729] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: LEARNER
I20240503 02:15:20.503211 13556 consensus_queue.cc:260] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } }
I20240503 02:15:20.503917 13556 ts_tablet_manager.cc:1398] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:15:20.504925 8972 tablet_copy_service.cc:342] P 99be4ff1114d4aaabcc537604fce54b3: Request end of tablet copy session 79e41f25066e4e5abbd71e48d545c6bf-64c52f9977ae4eb4ac803be99b6426c2 received from {username='slave'} at 127.23.104.196:50885
I20240503 02:15:20.505059 8972 tablet_copy_service.cc:433] P 99be4ff1114d4aaabcc537604fce54b3: ending tablet copy session 79e41f25066e4e5abbd71e48d545c6bf-64c52f9977ae4eb4ac803be99b6426c2 on tablet 64c52f9977ae4eb4ac803be99b6426c2 with peer 79e41f25066e4e5abbd71e48d545c6bf
I20240503 02:15:20.506778 13475 ts_tablet_manager.cc:1888] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:15:20.506994 13475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4679ab055c1a40f19bf7b602451044f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:20.510152 13475 tablet_copy_client.cc:783] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:20.510385 13475 tablet_copy_client.cc:647] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:20.513311 13475 tablet_copy_client.cc:523] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:20.517652 13475 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap starting.
I20240503 02:15:20.526741 13573 raft_consensus.cc:1050] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3: attempting to promote NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf to VOTER
I20240503 02:15:20.527909 13573 consensus_queue.cc:237] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: false } }
I20240503 02:15:20.534435 8821 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEARNER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.535378 10947 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.535956 9212 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.536497 10812 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.536679 13337 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.537312 13431 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.537400 9083 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.537895 13431 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.538483 13431 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.539148 13337 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.579989 13701 raft_consensus.cc:1050] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d: attempting to promote NON_VOTER 2a1893ab239f4163a06bdbf8ce579431 to VOTER
I20240503 02:15:20.581151 13701 consensus_queue.cc:237] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } }
W20240503 02:15:20.598372 8886 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.198:41237, user_credentials={real_user=slave}} blocked reactor thread for 60563.6us
W20240503 02:15:20.599066 8888 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.200:45027, user_credentials={real_user=slave}} blocked reactor thread for 61687.2us
I20240503 02:15:20.600064 8553 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.600930 8425 raft_consensus.cc:1261] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 99be4ff1114d4aaabcc537604fce54b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.601624 13431 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.632985 8553 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEARNER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.633766 13712 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.637625 10812 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.638535 13492 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.650378 11077 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.650460 8689 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.650621 13475 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:20.651052 13475 tablet_bootstrap.cc:492] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: Bootstrap complete.
I20240503 02:15:20.651141 13492 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.651324 13475 ts_tablet_manager.cc:1367] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent bootstrapping tablet: real 0.134s user 0.046s sys 0.005s
I20240503 02:15:20.651402 8421 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.651805 13492 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.652101 13475 raft_consensus.cc:348] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } }
I20240503 02:15:20.652324 13475 raft_consensus.cc:729] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 79e41f25066e4e5abbd71e48d545c6bf, State: Initialized, Role: LEARNER
I20240503 02:15:20.652580 13475 consensus_queue.cc:260] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { replace: true } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: NON_VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: true } }
I20240503 02:15:20.653280 13475 ts_tablet_manager.cc:1398] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:15:20.654069 8578 tablet_copy_service.cc:342] P 2a1893ab239f4163a06bdbf8ce579431: Request end of tablet copy session 79e41f25066e4e5abbd71e48d545c6bf-4679ab055c1a40f19bf7b602451044f6 received from {username='slave'} at 127.23.104.196:40793
I20240503 02:15:20.654237 8578 tablet_copy_service.cc:433] P 2a1893ab239f4163a06bdbf8ce579431: ending tablet copy session 79e41f25066e4e5abbd71e48d545c6bf-4679ab055c1a40f19bf7b602451044f6 on tablet 4679ab055c1a40f19bf7b602451044f6 with peer 79e41f25066e4e5abbd71e48d545c6bf
I20240503 02:15:20.654206 13492 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.673895 9212 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.674723 13712 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.676548 8819 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.677193 13492 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.690091 8950 raft_consensus.cc:1203] T 1a30758cbd1d4845bb9eeb02e1a2942a P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 02:15:20.694253 13474 raft_consensus.cc:1050] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865: attempting to promote NON_VOTER 99be4ff1114d4aaabcc537604fce54b3 to VOTER
I20240503 02:15:20.695663 13474 consensus_queue.cc:237] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: false } }
I20240503 02:15:20.703068 12585 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.703305 13736 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: false } } }
I20240503 02:15:20.705536 13735 consensus_queue.cc:1035] T a7a434e2df0a4ea0b2ab5c20034033c6 P 99be4ff1114d4aaabcc537604fce54b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.708995 13713 raft_consensus.cc:1050] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d: attempting to promote NON_VOTER dabeea74472746538132a520710248ea to VOTER
I20240503 02:15:20.709307 10814 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: false } } }
I20240503 02:15:20.709398 10947 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: false } } }
I20240503 02:15:20.710165 13713 consensus_queue.cc:237] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:20.712989 11076 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.716246 8273 catalog_manager.cc:5484] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4792082063bd4d58bc222c9c95b16335 reported cstate change: config changed from index 11 to 12, 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99be4ff1114d4aaabcc537604fce54b3" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: false } } }
I20240503 02:15:20.716441 10815 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.716390 8423 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.718761 8553 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.720261 8689 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.725203 8278 catalog_manager.cc:5484] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea reported cstate change: config changed from index 11 to 12, 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.733000 8950 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 99be4ff1114d4aaabcc537604fce54b3 [term 1 LEARNER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.731400 8821 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: false } } }
I20240503 02:15:20.733804 9083 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: false } } }
I20240503 02:15:20.744957 10947 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.745265 11076 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea [term 1 LEARNER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.745571 13492 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.747154 13761 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.749388 10814 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.750730 8689 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.751070 8553 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.751515 9213 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.751619 13492 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.752161 13492 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.752786 13492 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.753824 8950 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.757048 13712 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.761556 13712 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.767977 8820 raft_consensus.cc:1203] T 64c52f9977ae4eb4ac803be99b6426c2 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 02:15:20.777019 9210 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.787156 8425 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: false } } }
I20240503 02:15:20.811471 8950 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:20.811739 13713 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:20.816033 9081 consensus_queue.cc:237] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } }
I20240503 02:15:20.819476 11076 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.819649 8423 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.820257 13761 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.820647 12585 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.825410 8821 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.825510 10812 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.826131 13492 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.826618 13492 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.827620 8820 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 2a1893ab239f4163a06bdbf8ce579431 (127.23.104.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { replace: true } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.809542 8553 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: false } } }
I20240503 02:15:20.831626 8689 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:20.833067 10812 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:20.836249 9214 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:20.845352 8275 catalog_manager.cc:5484] T 5028c92b346845c38b3356a0d8c7507f P e512cb0d8f244ac6ab0e7150a0a8e2df reported cstate change: config changed from index 11 to 12, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:20.847435 8555 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:20.850416 12585 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.851804 8820 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.865746 8557 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.866568 13701 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.866180 10814 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.868502 8557 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.881173 13517 raft_consensus.cc:1050] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431: attempting to promote NON_VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 to VOTER
I20240503 02:15:20.882879 13517 consensus_queue.cc:237] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } }
I20240503 02:15:20.890257 13602 ts_tablet_manager.cc:902] T b71357f9f8ef449189f485b3246869f1 P 99be4ff1114d4aaabcc537604fce54b3: Initiating tablet copy from peer c399efcbba924863ac89d156dc07ac7d (127.23.104.198:41237)
I20240503 02:15:20.893828 13764 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:20.897497 8273 catalog_manager.cc:5484] T df68cc6032b14cae9e1cac074976baf8 P 79e41f25066e4e5abbd71e48d545c6bf reported cstate change: config changed from index 12 to 13, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New cstate: current_term: 1 leader_uuid: "c399efcbba924863ac89d156dc07ac7d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.919283 8670 tablet_service.cc:1514] Processing DeleteTablet for tablet df68cc6032b14cae9e1cac074976baf8 with delete_type TABLET_DATA_TOMBSTONED (TS 846e89dd0b154d2b9bb7d4f315c9fc12 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:36802
W20240503 02:15:20.921590 9149 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.104.197:45403, user_credentials={real_user=slave}} blocked reactor thread for 183771us
I20240503 02:15:20.923527 9211 raft_consensus.cc:2932] T a7a434e2df0a4ea0b2ab5c20034033c6 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 79e41f25066e4e5abbd71e48d545c6bf (127.23.104.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } attrs { replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } attrs { promote: false } } }
I20240503 02:15:20.927768 13789 tablet_replica.cc:330] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12: stopping tablet replica
I20240503 02:15:20.927990 13789 raft_consensus.cc:2229] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:20.930379 13789 raft_consensus.cc:2258] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:20.931994 8267 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df68cc6032b14cae9e1cac074976baf8 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:15:20.933051 13789 ts_tablet_manager.cc:1875] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:20.934881 10947 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:20.939725 8819 raft_consensus.cc:1203] T 4679ab055c1a40f19bf7b602451044f6 P 79e41f25066e4e5abbd71e48d545c6bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:15:20.941270 13789 ts_tablet_manager.cc:1888] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240503 02:15:20.941489 13789 log.cc:1196] T df68cc6032b14cae9e1cac074976baf8 P 846e89dd0b154d2b9bb7d4f315c9fc12: Deleting WAL directory at /tmp/dist-test-taskTHQSCq/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714702283285963-23971-0/raft_consensus-itest-cluster/ts-2/wal/wals/df68cc6032b14cae9e1cac074976baf8
I20240503 02:15:20.941850 9213 raft_consensus.cc:1261] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.942493 8267 catalog_manager.cc:4830] TS 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195:45495): tablet df68cc6032b14cae9e1cac074976baf8 (table rebalance_test_table_2 [id=68a41081d3814d0498a9c6bbfdc7a122]) successfully deleted
I20240503 02:15:20.943732 13712 consensus_queue.cc:1035] T df68cc6032b14cae9e1cac074976baf8 P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.945992 9213 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.951582 11077 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:20.951848 8425 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.959347 11076 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.960211 13033 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.960398 10944 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.962708 13033 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.968261 8689 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 LEARNER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.969074 13651 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.972160 8425 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 4c668b2e2d5c4edba46cbc5c36065c0c [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.972689 13651 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.978431 9083 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.979038 13651 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.991554 10812 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.992141 13517 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.994395 11077 raft_consensus.cc:2932] T df68cc6032b14cae9e1cac074976baf8 P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } attrs { promote: false } } }
I20240503 02:15:20.990403 8949 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:20.997071 13651 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:20.999151 13651 raft_consensus.cc:1013] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: attempt to promote peer 4792082063bd4d58bc222c9c95b16335: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 02:15:21.003042 11078 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P dabeea74472746538132a520710248ea [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, dabeea74472746538132a520710248ea (127.23.104.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:21.006876 9081 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P c399efcbba924863ac89d156dc07ac7d [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:21.008502 8685 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:21.008539 8555 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:21.009114 8819 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 79e41f25066e4e5abbd71e48d545c6bf [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:21.009495 13774 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240503 02:15:21.007777 10944 raft_consensus.cc:1261] T 00fc93702514499eb32902238c46356d P 4792082063bd4d58bc222c9c95b16335 [term 1 FOLLOWER]: Refusing update from remote peer cb90f34dc37b4b16b2a5b5a4ee012865: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:15:21.010763 13764 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:21.030555 9081 consensus_queue.cc:237] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } }
I20240503 02:15:21.035852 8952 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 99be4ff1114d4aaabcc537604fce54b3 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:21.036679 13712 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:21.038599 8685 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:21.039099 13761 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:21.040652 8555 raft_consensus.cc:1261] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Refusing update from remote peer c399efcbba924863ac89d156dc07ac7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:21.041755 13701 consensus_queue.cc:1035] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:21.045459 13713 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P c399efcbba924863ac89d156dc07ac7d [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:21.048817 8555 raft_consensus.cc:2932] T 5028c92b346845c38b3356a0d8c7507f P 2a1893ab239f4163a06bdbf8ce579431 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e512cb0d8f244ac6ab0e7150a0a8e2df (127.23.104.200) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } }
I20240503 02:15:21.051875 9213 raft_consensus.cc:1261] T 99e42dca2db54de5a9417df20242e2c2 P cb90f34dc37b4b16b2a5b5a4ee012865 [term 1 FOLLOWER]: Refusing update from remote peer 2a1893ab239f4163a06bdbf8ce579431: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:15:21.052438 13033 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:21.053992 13764 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e41f25066e4e5abbd71e48d545c6bf" member_type: VOTER last_known_addr { host: "127.23.104.196" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20240503 02:15:21.055177 13780 raft_consensus.cc:1050] T 879167ff15db41a7b4ef21fa10c44b27 P cb90f34dc37b4b16b2a5b5a4ee012865: attempting to promote NON_VOTER 79e41f25066e4e5abbd71e48d545c6bf to VOTER
I20240503 02:15:21.056651 13651 consensus_queue.cc:1035] T 99e42dca2db54de5a9417df20242e2c2 P 2a1893ab239f4163a06bdbf8ce579431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:15:21.058509 10946 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 4792082063bd4d58bc222c9c95b16335 [term 1 LEARNER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:21.059268 13775 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:21.058897 8689 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P 846e89dd0b154d2b9bb7d4f315c9fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: NON_VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: true } } }
I20240503 02:15:21.059844 13775 consensus_queue.cc:1035] T 00fc93702514499eb32902238c46356d P cb90f34dc37b4b16b2a5b5a4ee012865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792082063bd4d58bc222c9c95b16335" member_type: VOTER last_known_addr { host: "127.23.104.201" port: 46877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:15:21.059659 10814 raft_consensus.cc:2932] T 99e42dca2db54de5a9417df20242e2c2 P e512cb0d8f244ac6ab0e7150a0a8e2df [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 846e89dd0b154d2b9bb7d4f315c9fc12 (127.23.104.195) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "99be4ff1114d4aaabcc537604fce54b3" member_type: VOTER last_known_addr { host: "127.23.104.197" port: 45403 } } peers { permanent_uuid: "c399efcbba924863ac89d156dc07ac7d" member_type: VOTER last_known_addr { host: "127.23.104.198" port: 41237 } } peers { permanent_uuid: "cb90f34dc37b4b16b2a5b5a4ee012865" member_type: VOTER last_known_addr { host: "127.23.104.199" port: 32809 } } peers { permanent_uuid: "4c668b2e2d5c4edba46cbc5c36065c0c" member_type: VOTER last_known_addr { host: "127.23.104.193" port: 38635 } attrs { replace: true } } peers { permanent_uuid: "2a1893ab239f4163a06bdbf8ce579431" member_type: VOTER last_known_addr { host: "127.23.104.194" port: 45923 } } peers { permanent_uuid: "dabeea74472746538132a520710248ea" member_type: VOTER last_known_addr { host: "127.23.104.202" port: 37759 } attrs { promote: false } } peers { permanent_uuid: "e512cb0d8f244ac6ab0e7150a0a8e2df" member_type: VOTER last_known_addr { host: "127.23.104.200" port: 45027 } attrs { promote: false replace: true } } peers { permanent_uuid: "846e89dd0b154d2b9bb7d4f315c9fc12" member_type: VOTER last_known_addr { host: "127.23.104.195" port: 45495 } attrs { promote: false } } peers { permanent_uuid: "4792082063bd4d58bc222c9c95b16335"